Search

Nowadays many research articles are prefaced with research highlights to summarize the main findings of the paper. Highlights not only help researchers precisely and quickly identify the contributions of a paper, they also enhance the discoverability of the article via search engines. We aim to automatically construct research highlights given certain segments of a research paper. We use a pointer-generator network with coverage mechanism and a contextual embedding layer at the input that encodes the input tokens into SciBERT embeddings. We test our model on a benchmark dataset, CSPubSum, and also present MixSub, a new multi-disciplinary corpus of papers for automatic research highlight generation. For both CSPubSum and MixSub, we have observed that the proposed model achieves the best performance compared to related variants and other models proposed in the literature. On the CSPubSum dataset, our model achieves the best performance when the input is only the abstract of a paper as opposed to other segments of the paper. It produces ROUGE-1, ROUGE-2 and ROUGE-L F1-scores of 38.26, 14.26 and 35.51, respectively, METEOR score of 32.62, and BERTScore F1 of 86.65 which outperform all other baselines. On the new MixSub dataset, where only the abstract is the input, our proposed model (when trained on the whole training corpus without distinguishing between the subject categories) achieves ROUGE-1, ROUGE-2 and ROUGE-L F1-scores of 31.78, 9.76 and 29.3, respectively, METEOR score of 24.00, and BERTScore F1 of 85.25.

Music generation using deep learning has received considerable attention in recent years. Researchers have developed various generative models capable of imitating musical conventions, comprehending the musical corpora, and generating new samples based on the learning outcome. Although the samples generated by these models are persuasive, they often lack musical structure and creativity. For instance, a vanilla end-to-end approach, which deals with all levels of music representation at once, does not offer human-level control and interaction during the learning process, leading to constrained results. Indeed, music creation is a recurrent process that follows some principles by a musician, where various musical features are reused or adapted. On the other hand, a musical piece adheres to a musical style, breaking down into precise concepts of timbre style, performance style, composition style, and the coherency between these aspects. Here, we study and analyze the current advances in music generation using deep learning models through different criteria. We discuss the shortcomings and limitations of these models regarding interactivity and adaptability. Finally, we draw the potential future research direction addressing multi-agent systems and reinforcement learning algorithms to alleviate these shortcomings and limitations.

Due to the increasing size of integrated circuits (ICs), their design and optimization phases (i.e., computer-aided design, CAD) grow increasingly complex. At design time, a large design space needs to be explored to find an implementation that fulfills all specifications and then optimizes metrics like energy, area, delay, reliability, etc. At run time, a large configuration space needs to be searched to find the best set of parameters (e.g., voltage/frequency) to further optimize the system. Both spaces are infeasible for exhaustive search typically leading to heuristic optimization algorithms that find some tradeoff between design quality and computational overhead. Machine learning (ML) can build powerful models that have successfully been employed in related domains. In this survey, we categorize how ML may be used and is used for design-time and run-time optimization and exploration strategies of ICs. A metastudy of published techniques unveils areas in CAD that are well explored and underexplored with ML, as well as trends in the employed ML algorithms. We present a comprehensive categorization and summary of the state of the art on ML for CAD. Finally, we summarize the remaining challenges and promising open research directions. [ABSTRACT FROM AUTHOR]

Tracking the trends and taking early steps accordingly is important in academia, as well as in other domains such as technology and finance. In this work, we focus on the problem of predicting the trending research topics from a collection of academic papers. Previous efforts model the problem in different ways and mostly apply classical approaches such as correlation analysis and clustering. There are also several recent neural model based solutions, however they rely on feature vectors and additional information for the trend prediction. In this work, given a collection of publications within the observation time window, we predict whether the use of a keyword will increase, decrease or be steady for the future time window (prediction window). As the solution, we propose a family of deep neural architectures that focus on generating summary representations for paper collections under the query keyword. Due to the sequence based nature of the data, Long Short-Term Memory (LSTM) module plays a core role, but it is combined with different layers in a novel way. The first group of proposed neural architectures consider each paper as a sequence of keywords and use word embeddings to construct paper collection representations. In this group, the proposed architectures differ from each other in the way year based and overall summary representations are constructed. In the second group, each paper is directly represented as a vector and the use of different paper embedding techniques are explored. The analyses of the models are performed on a variety of paper collections belonging to different academic venues, obtained from Microsoft Academic Graph data set. The experiments conducted against baseline methods show that proposed deep neural based models achieve higher trend prediction performance than the baseline models on the overall. Among the proposed models, paper embedding based models provide better results for most of the cases.

The article reports that With the continued growth of IoT devices and their deployment, manually managing and connecting them is impractical and presents multiple challenges. To that end, Zero Touch Networks that rely on software-based modules instead of dedicated propriety hardware become a viable potential solution. The overall aim of zero-touch networks is for machines to learn how to become more autonomous so that we can delegate complex, mundane tasks to them.

This special section focuses on the state-of-the-art technologies on parallel and distributed computing techniques for artificial intelligence (AI), machine learning (ML), and deep learning (DL). AI, ML, and DL can enable computers the ability to learn from a large amount of data and use the learned model to optimize a complex problem or discover rules in a complicated system. AI, ML and DL can be applied to push forward the boundaries for many domains and significantly influence our daily life. [ABSTRACT FROM AUTHOR]

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research. [ABSTRACT FROM AUTHOR]

Visual surveillance for autonomous power facility inspection is considered to bethe prominent field of study in the power industry. This research field completely focuses on either object detection or image fusion which lacks the overall consideration. By considering this, a single end-to-end object detection method by incorporating the image fusion named Fast Detection Fusion Network (FDFNet) is proposed in this paper to output qualitative fused images with detection results. The parameters in the FDFnet are greatly reduced by sharing the feature extraction network between image fusion and object detection tasks, due to which a huge reduction in computational complexity is achieved. On this basis, the object detection algorithm performance on various types of power facility images is compared and analyzed. For experimentation purposes, an IR (infrared) and VIS (visible) image acquisition system has also been designed. In addition, a dataset named CVPower with different sets of images for power facility fusion detection is constructed for this research field. Experimental results demonstrate that the proposed method can achieve the mAP of not less than 70%, process 2 frames per second, and produce high qualitative fused images. [ABSTRACT FROM AUTHOR]

Detecting wafer map anomalies is crucial for preventing yield loss in semiconductor fabrication, although intricate patterns and resource-intensive labeled data prerequisites hinder precise deep-learning segmentation. This paper presents an innovative, unsupervised method for detecting pixel-level anomalies in wafer maps. It utilizes an efficient dual attention module with a knowledge distillation network to learn defect distributions without anomalies. Knowledge transfer is achieved by distilling information from a pre-trained teacher into a student network with similar architecture, except an efficient dual attention module is incorporated atop the teacher network’s feature pyramid hierarchies, which enhances feature representation and segmentation across pyramid hierarchies that selectively emphasize relevant and discard irrelevant features by capturing contextual associations in positional and channel dimensions. Furthermore, it enables student networks to acquire an improved knowledge of hierarchical features to identify anomalies across different scales accurately. The dissimilarity in feature pyramids acts as a discriminatory function, predicting the likelihood of an abnormality, resulting in highly accurate pixel-level anomaly detection. Consequently, our proposed method excelled on the WM-811K and MixedWM38 datasets, achieving AUROC, AUPR, AUPRO, and F1-Scores of (99.65%, 99.35%), (97.31%, 92.13%), (90.76%, 84.66%) respectively, alongside an inference speed of 3.204 FPS, showcasing its high precision and efficiency. [ABSTRACT FROM AUTHOR]

Multi-focus image fusion (MFIF) is an important area in image processing. Since 2017, deep learning has been introduced to the field of MFIF and various methods have been proposed. However, there is a lack of survey papers that discuss deep learning-based MFIF methods in detail. In this study, we fill this gap by giving a detailed survey on deep learning-based MFIF algorithms, including methods, datasets and evaluation metrics. To the best of our knowledge, this is the first survey paper that focuses on deep learning-based approaches in the field of MFIF. Besides, extensive experiments have been conducted to compare the performance of deep learning-based MFIF algorithms with conventional MFIF approaches. By analyzing qualitative and quantitative results, we give some observations on the current status of MFIF and discuss some future prospects of this field. [ABSTRACT FROM AUTHOR]

Unmanned Aerial Vehicle (UAV) inspection has gradually replaced manual inspection of transmission tower, which produces many images. While it is laborious and time-consuming to manually analyze these images, there are also challenges in automatically detecting small-size defective components such as bolts in transmission tower images, due to problems including complex background, small size, and many similar objects of bolts. In this paper, by virtue of multi-scale features and context information, we propose a deep neural network named Camp-Net (Context Information and Multi-Scale Pyramid Network) to identify bolts defect in transmission tower images. First, multi-scale feature fusion combines deep features and shallow features in convolutional networks to detect small-size bolts. Second, context information fusion puts the information around bolts into the detection network to remove the disturbance of complex background and similar objects. An image dataset containing defective bolts and normal bolts is constructed for model training and testing. Experimental results show that bolts with loose pins and bolts without pins among fittings in transmission tower can be accurately identified with the proposed model. The Average Precision (AP) of defective bolts detection of this model can be 11.4% higher than that of the commonly used high performance model, Faster R-CNN. [ABSTRACT FROM AUTHOR]

The IEEE Transactions on Semiconductor Manufacturing congratulates Haoyu Yang, Shuhe Li, Wen Chen, Piyush Pathak, Frank Gennari, Ya-Chieh Lai, and Bei Yu whose paper DeePattern: Layout Pattern Generation With Transforming Convolutional Auto-Encoder (Vol. 35, No.1, February 2022) was selected as the Best Paper for 2022 by a team of Associate Editors. This paper proposes generative machine learning models to synthesize VLSI layouts through a pattern generation framework that reduces the challenging creation problem into two simpler subproblems with the aid of an efficient squish pattern representation. The problem is treated by designing an innovative transforming convolutional auto-encoder in TCAE architecture, aiming to generate efficient and representative pattern shapes. The paper introduces a GAN-guided TCAE ML analysis that targets massive DRC-clean and content-specific pattern generation following certain design rules, without losing pattern library diversity. Through experiments on 7nm EUV designs, the authors demonstrate that each latent vector node in TCAE has a real physical meanings in layout domain and transformations on latent vectors can produce additional topologies of interest. Results show that the generated pattern library exhibits larger pattern number and pattern diversity compared to the traditional state-of-the-art industry layout generator. [ABSTRACT FROM AUTHOR]

In this paper, we present a comprehensive review of the imbalance problems in object detection. To analyze the problems in a systematic manner, we introduce a problem-based taxonomy. Following this taxonomy, we discuss each problem in depth and present a unifying yet critical perspective on the solutions in the literature. In addition, we identify major open issues regarding the existing imbalance problems as well as imbalance problems that have not been discussed before. Moreover, in order to keep our review up to date, we provide an accompanying webpage which catalogs papers addressing imbalance problems, according to our problem-based taxonomy. Researchers can track newer studies on this webpage available at: https://github.com/kemaloksuz/ObjectDetectionImbalance. [ABSTRACT FROM AUTHOR]

The explosive growth of Internet-of-Things (IoT) applications such as smart cities and Industry 4.0 have led to drastic increase in demand for wireless bandwidth, hence motivating the rapid development of new techniques for enhancing spectrum utilization needed by new generation wireless communication technologies. Among others, dynamic spectrum access (DSA) is one of the most widely accepted approaches. In this paper, as an enhancement of existing works, we take into consideration of inter-node collaborations in a dynamic spectrum environment. Typically, in such distributed circumstances, intelligent dynamic spectrum access almost invariably relies on self-learning to achieve dynamic spectrum access improvement. Whereas, this paper proposes a DSA scheme based on deep reinforcement learning to enhance spectrum and access efficiency. Unlike traditional Q-learning-based DSA, we introduce the following to enhance the spectrum efficiency in dynamic IoT spectrum environments. First, deep double Q-learning is adopted to perform local self-spectrum-learning for IoT terminals in order to achieve better dynamic access accuracy. Second, to accelerate learning convergence, federated learning (FL) in edge nodes is used to improve the self-learning. Third, multiple secondary users, who do not interfere with each other and have similar operation condition, are clustered for federated learning to enhance the efficiency of deep reinforcement learning. Comparing with the traditional distributed DSA with deep learning, the proposed scheme has faster access convergence speed due to the characteristic of global optimization for federated learning. Based on this, a framework of federated deep reinforcement learning (FDRL) for DSA is proposed. Furthermore, this scheme preserves privacy of IoT users in that FDRL only requires model parameters to be uploaded to edge servers. Simulations are performed to show the effectiveness of theproposed FDRL-based DSA framework. [ABSTRACT FROM AUTHOR]

Since gait can be utilized to identify individuals from a far distance without their interaction and coordination, recently many gait recognition methods have been proposed. However, due to a real-world scenario of clothing changes, a degradation occurs for most of these methods. Thus in this paper, a more efficient gait recognition method is proposed to address the problem of clothing variances. First, part-based gait features are formulated from two different perspectives, i.e., the separated body parts that are more robust to clothing changes and the estimated human skeleton key-point regions. It is reasonable to formulate such features for cloth-changing gait recognition, because these two perspectives are both less vulnerable to clothing changes. Given that each feature has its own advantages and disadvantages, a more efficient gait feature is generated in this paper by assembling these two features together. Moreover, since local features are more discriminative than global features, in this paper more attention is focused on the local short-range features. Also, unlike most methods, in our method we treat the estimated key-point features as a set of word embeddings, and a transformer encoder is specifically used to learn the dependence of each correlative key-points. The robustness and effectiveness of our proposed method are certified by experiments on CASIA Gait Dataset B, and it has achieved the state-of-the-art performance on this dataset. [ABSTRACT FROM AUTHOR]

The document is the table of contents for the May 2024 issue of the IEEE Transactions on Semiconductor Manufacturing. It includes papers on various topics such as yield modeling, analysis, and enhancement, advanced processing, advanced process control, equipment and automation technology, non-silicon materials, emerging areas, and environment, safety, and health. The articles cover subjects such as chip-scale chemical mechanical polishing, threshold voltage deviation monitoring, curvilinear standard cell design, fabrication of silicon nanocone arrays, improving the reliability of through silicon vias, energy consumption and carbon emission reduction in HVAC systems, low-k silicon dioxide synthesis, chemical mechanical polishing of diamond epitaxial layers, conditional variable selection based on deep learning, and eco-friendly dry-cleaning and diagnostics of silicon dioxide deposition chambers. The document also includes an announcement for the call for nominations for the 2024 IEEE EDS Early Career Award. [Extracted from the article]

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a best-response-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research. A list of important papers discussed in this survey, corresponding codes, and additional resources on BLOs are publicly available at: https://github.com/vis-opt-group/BLO. [ABSTRACT FROM AUTHOR]

Deep learning models have been shown to be vulnerable to adversarial attacks. Adversarial attacks are imperceptible perturbations added to an image such that the deep learning model misclassifies the image with a high confidence. Existing adversarial defenses validate their performance using only the classification accuracy. However, classification accuracy by itself is not a reliable metric to determine if the resulting image is “adversarial-free”. This is a foundational problem for online image recognition applications where the ground-truth of the incoming image is not known and hence we cannot compute the accuracy of the classifier or validate if the image is “adversarial-free” or not. This paper proposes a novel privacy preserving framework for defending Black box classifiers from adversarial attacks using an ensemble of iterative adversarial image purifiers whose performance is continuously validated in a loop using Bayesian uncertainties. The proposed approach can convert a single-step black box adversarial defense into an iterative defense and proposes three novel privacy preserving Knowledge Distillation (KD) approaches that use prior meta-information from various datasets to mimic the performance of the Black box classifier. Additionally, this paper proves the existence of an optimal distribution for the purified images that can reach a theoretical lower bound, beyond which the image can no longer be purified. Experimental results on six public benchmark datasets namely: 1) Fashion-MNIST, 2) CIFAR-10, 3) GTSRB, 4) MIO-TCD, 5) Tiny-ImageNet, and 6) MS-Celeb show that the proposed approach can consistently detect adversarial examples and purify or reject them against a variety of adversarial attacks. [ABSTRACT FROM AUTHOR]

Entity resolution (ER) identifies data instances referring to the same real-world entity and has received enormous research attention. In this paper, we examine the task of ER from a broader perspective, with its input extended from textual records, which are conventionally studied in the literature, to other modalities such as check-in sequences, GPS trajectories and surveillance video frames to generate new applications. Our goal in this paper is to design an effective model to uniformly support all these ER applications with different input formats. Technically, we fully exploit the semantic contexts of embedding vectors for the pair of input instances. In particular, we propose an integrated multi-context attention framework that takes into account self-attention, pair-attention and global-attention from three types of context. The idea can be further extended to incorporate attribute attention in order to support structured datasets. We conduct extensive experiments on a diverse class of entity resolutions tasks, including tasks on unstructured, structured and dirty textual records, check-in sequences, GPS trajectories and surveillance video frames. The experimental results verified the effectiveness and generality of our model. When compared with strong baselines in these applications, our model can achieve superior or comparative performance. [ABSTRACT FROM AUTHOR]

This paper introduces video domain generalization where most video classification networks degenerate due to the lack of exposure to the target domains of divergent distributions. We observe that the global temporal features are less generalizable, due to the temporal domain shift that videos from other unseen domains may have an unexpected absence or misalignment of the temporal relations. This finding has motivated us to solve video domain generalization by effectively learning the local-relation features of different timescales that are more generalizable, and exploiting them along with the global-relation features to maintain the discriminability. This paper presents the VideoDG framework with two technical contributions. The first is a new deep architecture named the Adversarial Pyramid Network, which improves the generalizability of video features by capturing the local-relation, global-relation, and cross-relation features progressively. On the basis of pyramid features, the second contribution is a new and robust approach of adversarial data augmentation that can bridge different video domains by improving the diversity and quality of augmented data. We construct three video domain generalization benchmarks in which domains are divided according to different datasets, different consequences of actions, or different camera views, respectively. VideoDG consistently outperforms the combinations of previous video classification models and existing domain generalization methods on all benchmarks. [ABSTRACT FROM AUTHOR]

Digital watermarking technology and watermark attack methods are mutually reinforcing and complementary. Currently, traditional watermark attack methods are relatively mature, but these traditional attack methods will inevitably damage the visual quality of original images (OIs). Therefore, this paper proposes a covert attack method called residual dense based imperceptible watermark attack network (RD-IWAN). First, this paper designs a watermark attack residual dense network (WARDN) based on the residual dense network (RDN), which can effectively remove the watermark information in the middle and low frequency features of the watermarked image (WMI). Second, to improve the attack ability of the network, this paper innovatively proposes a progressive preprocessing method based on the information enhancement preprocessing method. Concurrently, to ensure the imperceptibility of this watermark attack method, a comprehensive loss function that combines the perceptual loss and mean square error loss (MSE) of OI and attacked watermarked image (AWMI) is designed in this study. Finally, attack experiments are designed and performed on watermarks with different embedding strengths and sizes. Experimental results show that, compared to traditional attack methods, the watermark attack method proposed in this paper exhibits stronger attack ability and higher imperceptibility. [ABSTRACT FROM AUTHOR]

This paper proposes a novel scalable reinforcement learning approach for simultaneous routing and spectrum access in wireless ad-hoc networks. In most previous works on reinforcement learning for network optimization, the network topology is assumed to be fixed, and a different agent is trained for each transmission node—this limits scalability and generalizability. Further, routing and spectrum access are typically treated as separate tasks. Moreover, the optimization objective is usually a cumulative metric along the route, e.g., number of hops or delay. In this paper, we account for the physical-layer signal-to-interference-plus-noise ratio (SINR) in a wireless network and further show that bottleneck objective such as the minimum SINR along the route can also be optimized effectively using reinforcement learning. Specifically, we propose a scalable approach in which a single agent is associated with each flow and makes routing and spectrum access decisions as it moves along the frontier nodes. The agent is trained according to the physical-layer characteristics of the environment using a novel rewarding scheme based on the Monte Carlo estimation of the future bottleneck SINR. It learns to avoid interference by intelligently making joint routing and spectrum allocation decisions based on the geographical location information of the neighbouring nodes. [ABSTRACT FROM AUTHOR]

Data quality plays an important role during the training stage of machine/deep learning models. The annotation hinges on the experiences of domain experts. To acquire the expert’s knowledge in the context of machine learning, manual data labeling, a tedious and time-consuming task in supervised learning, should be given a top priority. However, the domain experts in the line of plentiful manual annotation may easily get distracted or fatigued after long-time work, causing judgment errors, mislabeling, etc. The pattern recognition of wafer defect map is investigated in this paper, the primary goal of which is to train the convolutional neural network (CNN) model through a very limited number of manually labeled data so that the trained model is capable of performing pseudo labeling. Subsequently, a self-assured adaptive ensemble learner in terms of a series of shallow neural networks is proposed to filter wafer map samples with untrusted pseudo-labels. In the result, the amount of human annotations is significantly reduced by 61% for training a highly accurate classifier. A minimum number of manually labeled data is suggested while the equally high classification performance of wafer defect pattern is maintained. For the evaluation purpose, the proposed self-assured learning is compared with the confidence learning. [ABSTRACT FROM AUTHOR]

Radio signal recognition has a wide application in future communication systems and the vehicular communication, whose core is the extraction of signal features such as electromagnetic fingerprints. With the rapid development of artificial intelligence technology, deep learning has made amazing breakthroughs in image recognition, speech recognition and other fields. Deep learning is applied to electromagnetic fingerprint extraction in this paper. Firstly, thousands of the downlink aircraft communications addressing and reporting system (ACARS) signals used for communication between civil aircraft and airport tower are collected and generated. Then a pre-transformation network suitable for electromagnetic signals is constructed to convert one-dimensional signals into two-dimensional feature maps, and afterwards the feature maps are input into the convolutional neural network (CNN) for classification. By adopting the attention modules, the classification results were improved by a few percentage points over the baseline with a little cost. The method proposed in this paper achieves an accuracy rate of 94.1% and can obtain the aircraft type in a shorter time than traditional method. Moreover, the robustness of the proposed model in response to additive Gaussian white noise (AWGN) and phase deviation is studied and tested. [ABSTRACT FROM AUTHOR]

Link prediction aims at inferring missing links or predicting future ones based on the currently observed network. This topic is important for many applications such as social media, bioinformatics and recommendation systems. Most existing methods focus on homogeneous settings and consider only low-order pairwise relations while ignoring either the heterogeneity or high-order complex relations among different types of nodes, which tends to lead to a sub-optimal embedding result. This paper presents a method named Heterogeneous Hypergraph Variational Autoencoder (HeteHG-VAE) for link prediction in heterogeneous information networks (HINs). It first maps a conventional HIN to a heterogeneous hypergraph with a certain kind of semantics to capture both the high-order semantics and complex relations among nodes, while preserving the low-order pairwise topology information of the original HIN. Then, deep latent representations of nodes and hyperedges are learned by a Bayesian deep generative framework from the heterogeneous hypergraph in an unsupervised manner. Moreover, a hyperedge attention module is designed to learn the importance of different types of nodes in each hyperedge. The major merit of HeteHG-VAE lies in its ability of modeling multi-level relations in heterogeneous settings. Extensive experiments on real-world datasets demonstrate the effectiveness and efficiency of the proposed method. [ABSTRACT FROM AUTHOR]

With the development of digital medical technology, ubiquitous smartphones are emerging as valuable tools for the detection of complex and elusive diseases. This paper exploits smartphone walking recording for early detection of Parkinson's disease (PD) and finds that walking recording empowered by deep learning is a valid digital biomarker for early-recognizing PD patients. Specifically, the inertial sensor data is preprocessed, including normalization, scaling, and rotation, and then the processed data is fed into the proposed deep NeuroEnhanceNet. Finally, determine the individual prediction score using the PD-prone strategy and generate the detection results. The proposed deep NeuroEnhanceNet, specifically designed for inertial sensor data, can focus on both the long-term data characteristics within a single channel and the inter-channel correlations. Our method obtains a low false negative rate of 0.053 for the early detection of PD. We further analyze and compare the effectiveness of digital biomarkers captured from the walking and resting processes for early detection of PD. All the code for this work is available at: https://github.com/heyiyia/NeuroEnhanceNet.

Training an accurate classifier for EEG-based brain-computer interface (BCI) requires EEG data from a large number of users, whereas protecting their data privacy is a critical consideration. Federated learning (FL) is a promising solution to this challenge. This paper proposes Federated classification with local Batch-specific batch normalization and Sharpness-aware minimization (FedBS) for privacy protection in EEG-based motor imagery (MI) classification. FedBS utilizes local batch-specific batch normalization to reduce data discrepancies among different clients, and sharpness-aware minimization optimizer in local training to improve model generalization. Experiments on three public MI datasets using three popular deep learning models demonstrated that FedBS outperformed six state-of-the-art FL approaches. Remarkably, it also outperformed centralized training, which does not consider privacy protection at all. In summary, FedBS protects user EEG data privacy, enabling multiple BCI users to participate in large-scale machine learning model training, which in turn improves the BCI decoding accuracy.

Sleep staging serves as a fundamental assessment for sleep quality measurement and sleep disorder diagnosis. Although current deep learning approaches have successfully integrated multimodal sleep signals, enhancing the accuracy of automatic sleep staging, certain challenges remain, as follows: 1) optimizing the utilization of multi-modal information complementarity, 2) effectively extracting both long- and short-range temporal features of sleep information, and 3) addressing the class imbalance problem in sleep data. To address these challenges, this paper proposes a two-stream encode-decoder network, named TSEDSleepNet, which is inspired by the depth sensitive attention and automatic multi-modal fusion (DSA2F) framework. In TSEDSleepNet, a two-stream encoder is used to extract the multiscale features of electrooculogram (EOG) and electroencephalogram (EEG) signals. And a self-attention mechanism is utilized to fuse the multiscale features, generating multi-modal saliency features. Subsequently, the coarser-scale construction module (CSCM) is adopted to extract and construct multi-resolution features from the multiscale features and the salient features. Thereafter, a Transformer module is applied to capture both long- and short-range temporal features from the multi-resolution features. Finally, the long- and short-range temporal features are restored with low-layer details and mapped to the predicted classification results. Additionally, the Lovász loss function is applied to alleviate the class imbalance problem in sleep datasets. Our proposed method was tested on the Sleep-EDF-39 and Sleep-EDF-153 datasets, and it achieved classification accuracies of 88.9% and 85.2% and Macro-F1 scores of 84.8% and 79.7%, respectively, thus outperforming conventional traditional baseline models. These results highlight the efficacy of the proposed method in fusing multi-modal information. This method has potential for application as an adjunct tool for diagnosing sleep disorders.

Automatic identification of visual learning style in real time using raw electroencephalogram (EEG) is challenging. In this work, inspired by the powerful abilities of deep learning techniques, deep learning-based models are proposed to learn high-level feature representation for EEG visual learning identification. Existing computer-aided systems that use electroencephalograms and machine learning can reasonably assess learning styles. Despite their potential, offline processing is often necessary to eliminate artifacts and extract features, making these methods unsuitable for real-time applications. The dataset was chosen with 34 healthy subjects to measure their EEG signals during resting states (eyes open and eyes closed) and while performing learning tasks. The subjects displayed no prior knowledge of the animated educational content presented in video format. The paper presents an analysis of EEG signals measured during a resting state with closed eyes using three deep learning techniques: Long-term, short-term memory (LSTM), Long-term, short-term memory-convolutional neural network (LSTM-CNN), and Long-term, short-term memory-Fully convolutional neural network (LSTM-FCNN). The chosen techniques were based on their suitability for real-time applications with varying data lengths and the need for less computational time. The optimization of hypertuning parameters has enabled the identification of visual learners through the implementation of three techniques. LSTM-CNN technique has the highest average accuracy of 94%, a sensitivity of 80%, a specificity of 92%, and an F1 score of 94% when identifying the visual learning style of the student out of all three techniques. This research has shown that the most effective method is the deep learning-based LSTM-CNN technique, which accurately identifies a student's visual learning style.

Deep learning methods have advanced quickly in brain imaging analysis over the past few years, but they are usually restricted by the limited labeled data. Pre-trained model on unlabeled data has presented promising improvement in feature learning in many domains, such as natural language processing. However, this technique is under-explored in brain network analysis. In this paper, we focused on pre-training methods with Transformer networks to leverage existing unlabeled data for brain functional network classification. First, we proposed a Transformer-based neural network, named as BrainNPT, for brain functional network classification. The proposed method leveraged token as a classification embedding vector for the Transformer model to effectively capture the representation of brain networks. Second, we proposed a pre-training framework for BrainNPT model to leverage unlabeled brain network data to learn the structure information of brain functional networks. The results of classification experiments demonstrated the BrainNPT model without pre-training achieved the best performance with the state-of-the-art models, and the BrainNPT model with pre-training strongly outperformed the state-of-the-art models. The pre-training BrainNPT model improved 8.75% of accuracy compared with the model without pre-training. We further compared the pre-training strategies and the data augmentation methods, analyzed the influence of the parameters of the model, and explained the trained model.

Early diagnosis and intervention of depression promote complete recovery, with its traditional clinical assessments depending on the diagnostic scales, clinical experience of doctors and patient cooperation. Recent researches indicate that functional near-infrared spectroscopy (fNIRS) based on deep learning provides a promising approach to depression diagnosis. However, collecting large fNIRS datasets within a standard experimental paradigm remains challenging, limiting the applications of deep networks that require more data. To address these challenges, in this paper, we propose an fNIRS-driven depression recognition architecture based on cross-modal data augmentation (fCMDA), which converts fNIRS data into pseudo-sequence activation images. The approach incorporates a time-domain augmentation mechanism, including time warping and time masking, to generate diverse data. Additionally, we design a stimulation task-driven data pseudo-sequence method to map fNIRS data into pseudo-sequence activation images, facilitating the extraction of spatial-temporal, contextual and dynamic characteristics. Ultimately, we construct a depression recognition model based on deep classification networks using the imbalance loss function. Extensive experiments are performed on the two-class depression diagnosis and five-class depression severity recognition, which reveal impressive results with accuracy of 0.905 and 0.889, respectively. The fCMDA architecture provides a novel solution for effective depression recognition with limited data.

Major Depression Disorder (MDD) is a common yet destructive mental disorder that affects millions of people worldwide. Making early and accurate diagnosis of it is very meaningful. Recently, EEG, a non-invasive technique of recording spontaneous electrical activity of brains, has been widely used for MDD diagnosis. However, there are still some challenges in data quality and data size of EEG: (1) A large amount of noise is inevitable during EEG collection, making it difficult to extract discriminative features from raw EEG; (2) It is difficult to recruit a large number of subjects to collect sufficient and diverse data for model training. Both of the challenges cause the overfitting problem, especially for deep learning methods. In this paper, we propose DiffMDD, a diffusion-based deep learning framework for MDD diagnosis using EEG. Specifically, we extract more noise-irrelevant features to improve the model's robustness by designing the Forward Diffusion Noisy Training Module. Then we increase the size and diversity of data to help the model learn more generalized features by designing the Reverse Diffusion Data Augmentation Module. Finally, we re-train the classifier on the augmented dataset for MDD diagnosis. We conducted comprehensive experiments to test the overall performance and each module's effectiveness. The framework was validated on two public MDD diagnosis datasets, achieving the state-of-the-art performance.

The olfactory system enables humans to smell different odors, which are closely related to emotions. The high temporal resolution and non-invasiveness of Electroencephalogram (EEG) make it suitable to objectively study human preferences for odors. Effectively learning the temporal dynamics and spatial information from EEG is crucial for detecting odor-induced emotional valence. In this paper, we propose a deep learning architecture called Temporal Attention with Spatial Autoencoder Network (TASA) for predicting odor-induced emotions using EEG. TASA consists of a filter-bank layer, a spatial encoder, a time segmentation layer, a Long Short-Term Memory (LSTM) module, a multi-head self-attention (MSA) layer, and a fully connected layer. We improve upon the previous work by utilizing a two-phase learning framework, using the autoencoder module to learn the spatial information among electrodes by reconstructing the given input with a latent representation in the spatial dimension, which aims to minimize information loss compared to spatial filtering with CNN. The second improvement is inspired by the continuous nature of the olfactory process; we propose to use LSTM-MSA in TASA to capture its temporal dynamics by learning the intercorrelation among the time segments of the EEG. TASA is evaluated on an existing olfactory EEG dataset and compared with several existing deep learning architectures to demonstrate its effectiveness in predicting olfactory-triggered emotional responses. Interpretability analyses with DeepLIFT also suggest that TASA learns spatial-spectral features that are relevant to olfactory-induced emotion recognition.

Computational biomechanical analysis plays a pivotal role in understanding and improving human movements and physical functions. Although physics-based modeling methods can interpret the dynamic interaction between the neural drive to muscle dynamics and joint kinematics, they suffer from high computational latency. In recent years, data-driven methods have emerged as a promising alternative due to their fast execution speed, but label information is still required during training, which is not easy to acquire in practice. To tackle these issues, this paper presents a novel physics-informed deep learning method to predict muscle forces without any label information during model training. In addition, the proposed method could also identify personalized muscle-tendon parameters. To achieve this, the Hill muscle model-based forward dynamics is embedded into the deep neural network as the additional loss to further regulate the behavior of the deep neural network. Experimental validations on the wrist joint from six healthy subjects are performed, and a fully connected neural network (FNN) is selected to implement the proposed method. The predicted results of muscle forces show comparable or even lower root mean square error (RMSE) and higher coefficient of determination compared with baseline methods, which have to use the labeled surface electromyography (sEMG) signals, and it can also identify muscle-tendon parameters accurately, demonstrating the effectiveness of the proposed physics-informed deep learning method.

Brain-computer interfaces (BCIs) can enable direct communication with assistive devices by recording and decoding signals from the brain. To achieve high performance, many electrodes will be used, such as the recently developed invasive BCIs with channel numbers up to hundreds or even thousands. For those high-throughput BCIs, channel selection is important to reduce signal redundancy and invasiveness while maintaining decoding performance. However, such endeavour is rarely reported for invasive BCIs, especially those using deep learning methods. Two deep learning-based methods, referred to as Gumbel and STG, were proposed in this paper. They were evaluated using the Stereo-electroencephalography (SEEG) signals, and compared with three other methods, including manual selection, mutual information-based method (MI), and all channels (all channels without selection). The task is to classify the SEEG signals into five movements using channels selected by each method. When 10 channels were selected, the mean classification accuracies using Gumbel, STG (referred to as STG-10), manual selection, and MI selection were 65%, 60%, 60%, and 47%, respectively, whilst the accuracy was 59% using all channels (no selection). In addition, an investigation of the selected channels showed that Gumbel and STG have successfully identified the pre-central and post-central areas, which are closely related to motor control. Both Gumbel and STG successfully selected the informative channels in SEEG recordings while maintaining decoding accuracy. This study enables future high-throughput BCIs using deep learning methods, to identify useful channels and reduce computing and wireless transmission pressure.

The vanilla Few-shot Learning (FSL) learns to build a classifier for a new concept from one or very few target examples, with the general assumption that source and target classes are sampled from the same domain. Recently, the task of Cross-Domain Few-Shot Learning (CD-FSL) aims at tackling the FSL where there is a huge domain shift between the source and target datasets. Extensive efforts on CD-FSL have been made via either directly extending the meta-learning paradigm of vanilla FSL methods, or employing massive unlabeled target data to help learn models. In this paper, we notice that in the CD-FSL task, the few labeled target images have never been explicitly leveraged to inform the model in the training stage. However, such a labeled target example set is very important to bridge the huge domain gap. Critically, this paper advocates a more practical training scenario for CD-FSL. And our key insight is to utilize a few labeled target data to guide the learning of the CD-FSL model. Technically, we propose a novel Generalized Meta-learning based Feature-Disentangled Mixup network, namely GMeta-FDMixup. We make three key contributions of utilizing GMeta-FDMixup to address CD-FSL. Firstly, we present two mixup modules – mixup-P and mixup-M that help facilitate utilizing the unbalanced and disjoint source and target datasets. These two novel modules enable diverse image generation for training the model on the source domain. Secondly, to narrow the domain gap explicitly, we contribute a novel feature disentanglement module that learns to decouple the domain-irrelevant and domain-specific features. By stripping the domain-specific features, we alleviate the negative effects caused by the domain inductive bias. Finally, we repurpose a new contrastive learning module, dubbed ConL. ConL prevents the model from only capturing category-related features via introducing contrastive loss. Thus, the generalization ability on novel categories is improved. Extensive experimental results on two benchmarks show the superiority of our setting and the effectiveness of our method. Code and models will be released. [ABSTRACT FROM AUTHOR]

Multiview video coding (MVC) aims to compress the multiview video through the elimination of video redundancies, where the quality of the reference frame directly affects the compression efficiency. In this paper, we propose a deep virtual reference frame generation method based on a disparity-aware reference frame generation network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. The proposed DAG-Net consists of a multi-level receptive field module, a disparity-aware alignment module, and a fusion reconstruction module. First, a multi-level receptive field module is designed to enlarge the receptive field, and extract the multi-scale deep features of the temporal and inter-view reference frames. Then, a disparity-aware alignment module is proposed to learn the disparity relationship, and perform disparity shift on the inter-view reference frame to align it with the temporal reference frame. Finally, a fusion reconstruction module is utilized to fuse the complementary information and generate a more reliable virtual reference frame. Experiments demonstrate that the proposed reference frame generation method achieves superior performance for multiview video coding. [ABSTRACT FROM AUTHOR]

This paper investigates the optimal signal detection problem with a particular interest in large-scale multiple-input multiple-output (MIMO) systems. The problem is NP-hard and can be solved optimally by searching the shortest path on the decision tree. Unfortunately, the existing optimal search algorithms often involve prohibitively high complexities, which indicates that they are infeasible in large-scale MIMO systems. To address this issue, we propose a general heuristic search algorithm, namely, hyper-accelerated tree search (HATS) algorithm. The proposed algorithm employs a deep neural network (DNN) to estimate the optimal heuristic, and then use the estimated heuristic to speed up the underlying memory-bounded search algorithm. This idea is inspired by the fact that the underlying heuristic search algorithm reaches the optimal efficiency with the optimal heuristic function. Simulation results show that the proposed algorithm reaches almost the optimal bit error rate (BER) performance in large-scale systems, while the memory size can be bounded. In the meanwhile, it visits nearly the fewest tree nodes. This indicates that the proposed algorithm reaches almost the optimal efficiency in practical scenarios, and thereby it is applicable for large-scale systems. Besides, the code for this paper is available at https://github.com/skypitcher/hats. [ABSTRACT FROM AUTHOR]

Ride-sourcing services are now reshaping the way people travel by effectively connecting drivers and passengers through mobile internets. Online matching between idle drivers and waiting passengers is one of the most key components in a ride-sourcing system. The average pickup distance or time is an important measurement of system efficiency since it affects both passengers’ waiting time and drivers’ utilization rate. It is naturally expected that a more effective bipartite matching (with smaller average pickup time) can be implemented if the platform accumulates more idle drivers and waiting passengers in the matching pool. A specific passenger request can also benefit from a delayed matching since he/she may be matched with closer idle drivers after waiting for a few seconds. Motivated by the potential benefits of delayed matching, this paper establishes a two-stage framework which incorporates a combinatorial optimization and multi-agent deep reinforcement learning methods. The multi-agent reinforcement learning methods are used to dynamically determine the delayed time for each passenger request (or the time at which each request enters the matching pool), while the combinatorial optimization conducts an optimal bipartite matching between idle drivers and waiting passengers in the matching pool. Four tailored reinforcement learning methods, delayed multi-agent deep Q learning (Delayed-M-DQN), delayed multi-agent actor-critic (Delayed-M-A2C), delayed multi-agent Proximal Policy Optimization (Delayed-M-PPO), and delayed multi-agent actor-critic with experience replay (Delayed-M-ACER), are developed. Through extensive empirical experiments with a well-designed simulator, we show that the proposed framework is able to remarkably improve system performances, by well balancing the trade-off among pick-up time, matching time, successful matching rate. [ABSTRACT FROM AUTHOR]

This paper proposes a deep reinforcement learning-based video streaming scheme for mobility-aware vehicular networks, e.g., vehicles on the highway. We consider infrastructure-assisted and mmWave-based scenarios in which the macro base station (MBS) cannot directly provide the streaming service to vehicles due to the short range of mmWave beams so that small mmWave base stations (mBSs) along the road deliver the desired videos to users. Motivated by the fact that a video stream consists of sequential chunks and smooth streaming has to be provided while users pass through multiple mBSs at high speed, the MBS proactively pushes some video chunks of the desired contents to mBSs. This is done to support vehicles that are currently covered and/or will be by each mBS. We formulate the dynamic video delivery scheme that adaptively determines 1) which content, 2) what quality and 3) how many chunks to be proactively delivered from the MBS to mBSs using Markov decision process (MDP). Since it is difficult for the MBS to track all the channel conditions and the network states have extensive dimensions, we adopt the deep deterministic policy gradient (DDPG) algorithm for the DRL-based video delivery scheme. This paper finally shows that the DRL agentlearns a streaming policy that pursues high average quality while limiting packet drops, avoiding playback stalls, reducing quality fluctuations and saving backhaul usage. [ABSTRACT FROM AUTHOR]

An editorial is presented on addressing the need for bringing together leading efforts in non-Euclidean deep learning across all communities. Topics include Convolutional Neural Networks (CNNs) relying on classical signal processing models limiting the applicability to data with underlying Euclidean grid-like structure; and multiple levels of abstraction being introduced similarly to multi-layer networks.

As the semiconductor manufacturing process is moving towards the 3 nm node, there is a crucial need to reduce the edge placement error (EPE) to ensure proper functioning of the integrated circuit (IC) devices. EPE is the most important metric that quantify the fidelity of fabricated patterns in multi-patterning processes, and it is the combination of overlay errors and critical dimension (CD) errors. Recent advances in machine learning have enabled many new possibilities to improve the performance and efficiency of EPE optimization techniques. In this paper, we conducted a survey of recent research work that applied machine learning/ deep learning techniques for the purposes of enhancing virtual overlay metrology, reducing overlay error, and improving mask optimization methods for EPE reduction. Thorough discussions about the objectives, datasets, input features, models, key findings, and limitations are provided. In general, the results of the review work show a great potential of machine learning techniques in aiding the improvement of EPE in the field of semiconductor manufacturing. [ABSTRACT FROM AUTHOR]

Traditional Artificial Intelligence (AI) based fault detection approaches need a large amount of data for the model learning. However, in a real-world system, it is very difficult and expensive to obtain massive labeled fault data. In addition, the working conditions of a motor are usually variable, conventional fault diagnosis models with weak generalization ability can only be used for fault detection under constant working condition. The performance of traditional AI based approaches decreases when the working condition changes. To this end, a novel deep Gaussian process (GP) kernel transfer based few-shot learning method (RNGPT) is proposed in this paper for the fault detection of electric machines. First, a deep residual network (ResNet) is used to extract the features of the raw data. Then, the encoded latent feature vector is fed into the GP with kernel transfer ability to make the motor fault detection and classification. The proposed method uses much less data than the traditional AI based method to achieve fault diagnosis under variable working condition, and does not cause an overfitting problem. Experimental results of two case studies demonstrate that the proposed RNGPT model can accurately and effectively detect motor faults with limited labeled data under different working conditions. Experimental results of RNGPT with radial basis function (RBF) kernel model on simulation data present that the fault detection accuracy of the proposed method is about 16% higher than the conventional deep learning methods, 6% higher than other few-shot learning based methods in 5-shot and 4% higher in 1-shot. Finally, experimental on a real-world dataset, the RNGPT-RBF model still has the highest fault diagnosis accuracy in 5-shot (99.39 $ \pm $ 0.09%) and 1-shot (98.55 $ \pm $ 0.16%). [ABSTRACT FROM AUTHOR]

As a follow-up to the first IEEE Transactions on Medical Imaging (TMI) special issue on the theme of deep tomographic reconstruction, the second special issue is assembled to reflect the status and momentum of this rapidly emerging field. In this editorial, we provide a brief background illustrating the motivation for the development of network-based, data-driven, and learning-oriented reconstruction methods, summarize the included papers, and report our verification of the shared deep learning codes. Finally, we discuss several important research topics to facilitate further investigation and collaboration. [ABSTRACT FROM AUTHOR]

Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the schema and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used datasets for images, videos, audios, and 3D data, as well as the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning. [ABSTRACT FROM AUTHOR]