Search

Heterogeneous domain adaptation (HDA) tackles the learning of cross-domain samples with both different probability distributions and feature representations. Most of the existing HDA studies focus on the single-source scenario. In reality, however, it is not uncommon to obtain samples from multiple heterogeneous domains. In this article, we study the multisource HDA problem and propose a conditional weighting adversarial network (CWAN) to address it. The proposed CWAN adversarially learns a feature transformer, a label classifier, and a domain discriminator. To quantify the importance of different source domains, CWAN introduces a sophisticated conditional weighting scheme to calculate the weights of the source domains according to the conditional distribution divergence between the source and target domains. Different from existing weighting schemes, the proposed conditional weighting scheme not only weights the source domains but also implicitly aligns the conditional distributions during the optimization process. Experimental results clearly demonstrate that the proposed CWAN performs much better than several state-of-the-art methods on four real-world datasets.

Time series anomaly detection (TSAD) is an essential problem faced in several fields, e.g. fault detection, fraud detection, and intrusion detection, etc. Although TSAD is a crucial problem in anomaly detection, few solutions in anomaly detection are suitable for it at present. Recently, some researchers use GAN-based methods such as TAnoGAN and TadGAN to solve TSAD problem. However, problems such as model collapse, low generalization capability and poor accuracy still exist. In this paper, we proposed a Dilated Convolutional Transformer-based GAN (DCT-GAN) to enhance accuracy and improve generalization capability of the model. Specifically, DCT-GAN utilize several generators and a single discriminator to alleviate the mode collapse problem. Each generator consists of a dilated convolutional neural network and a Transformer block to obtain fine-grained and coarse-grained information of the time series, which is a useful component to improve generalization capability. We also use weight-based mechanism to balance these generators. Experiments verify the effectiveness of our method and each part of DCT-GAN.

Rumors can cause devastating consequences to individuals and our society. Analysis shows that the widespread of rumors typically results from deliberate promotion of information aiming to shape the collective public opinions on the concerned event. In this paper, we combat such chaotic phenomenon with a countermeasure by mirroring against how such chaos is created to make rumor detection more robust and effective. Our idea is inspired by adversarial learning method originated from Generative Adversarial Networks (GAN). We propose a GAN-style approach, where a generator is designed to produce uncertain or conflicting voices, further polarizing the original conversational threads to boost the discriminator. We reveal that feature learning effectiveness is highly relevant to the quality of generated parody. Given the strong natural language generation performance of transformer, we propose a transformer-based method to improve the generated posts, which appear to be closely responsive to the source post and retain the authentic propagation structure. Different from traditional data-driven rumor detection approaches, our method can capture low-frequency but more salient non-trivial discriminant patterns. Extensive experiments on THREE benchmarks demonstrate that our rumor detection method achieves much better results than state-of-the-art methods, and the transformer-based model further improve the performance of our GAN-style approach.

Recent transformer-based models, especially patch-based methods, have shown huge potentiality in vision tasks. However, the split fixed-size patches divide the input features into the same size patches, which ignores the fact that vision elements are often various and thus may destroy the semantic information. Also, the vanilla patch-based transformer cannot guarantee the information communication between patches, which will prevent the extraction of attention information with a global view. To circumvent those problems, we propose the Efficient Attention Pyramid Transformer (EAPT) in this paper. More specifically, we first propose Deformable Attention, which learns an offset for each position in patches. Therefore, even with split fixed-size patches, our method can still obtain non-fixed attention information that can cover various vision elements. Then, we design the Encode-Decode Communication module (En-DeC module), which can obtain communication information between all patches to get more complete global attention information. Finally, we also proposed a position encoding specifically for vision transformers, which can be used for patches of any dimensions and any lengths. Extensive experiments on the vision tasks of image classification, object detection, semantic segmentation demonstrate the effectiveness of our proposed model. Furthermore, we also conduct rigorous ablation studies to evaluate the key components of the proposed structure.

The objective of this paper is to propose a new inductorless bidirectional switched-capacitor power electronics transformer (IBSCX) for low-power commercial and residential applications. The proposed IBSCX is composed of multiple switched-capacitor converters (SCCs) and two bidirectional unfolders. Compared to the traditional solution which is the employment of a low-frequency transformer, the advantages of the proposed IBSCX include control circuit simplicity, high conversion efficiency, and volume and weight reduction for low power applications. The principle of operation, voltage gain analysis, and design methodology are described in this paper. Also, with the interleaved operation between SCCs, the output current ripple can be reduced. Finally, the computer simulations and hardware experimental results are shown to verify the performance of the proposed IBSCX. A prototype IBSCX is built and tested, and the peak and rated power conversion efficiency are 98.26% and 97.95%, respectively.

Recently, IGCTs have shown a great potential for the soft-switching dc transformer applications based on LLC resonant converter. The zero-voltage switching that is present in this converter topology enables an operation of IGCTs without clamping circuit and with significantly higher switching frequencies, while maintaining the superiorly low conduction losses. The high voltage and current ratings of the IGCTs make them a preferred choice for an efficient bulk power transmission. This paper shows how gate units for IGCTs can be optimized for this soft-switching application. It presents a simplification of the driving topology and the design process for high switching frequencies. This results in low power consumption and a very small size of the gate unit. The prototype of the gate unit is built and experimentally validated in a 2.5 kV 0.75 kA resonant operation of the IGCT.

Traffic forecasting has attracted considerable attention due to its importance in proactive urban traffic control and management. Scholars and engineers have exerted considerable efforts in improving the performance of traffic forecasting algorithms in terms of accuracy, reliability, and efficiency. Spatial feature representation of traffic flow is a core component that greatly influences traffic forecasting performance. In previous studies, several spatial attributes of traffic flow are ignored due to the following issues: a) traffic flow propagation does not comply with the road network, b) the spatial pattern of traffic flow varies over time, and c) single adjacent matrix cannot handle the complex and hierarchical urban traffic flow. To address the abovementioned issues, this study proposes a novel traffic forecasting algorithm called traffic transformer, which achieves great success in natural language processing. The multihead attention mechanism and stacking layers enable the transformer to learn dynamic and hierarchical features in sequential data. Two components, namely, global encoder and global-local decoder, are proposed to extract and fuse the spatial patterns globally and locally. Experimental results indicate that the proposed traffic transformer outperforms state-of-the-art methods. The learned dynamic and hierarchical features of traffic flow can help achieve a better understanding of spatial dependency of traffic flow for effective and efficient traffic control and management strategies.

After more than 70 years of evolution, great achievements have been made in machine translation. Especially in recent years, translation quality has been greatly improved with the emergence of neural machine translation (NMT). In this article, we first review the history of machine translation from rule-based machine translation to example-based machine translation and statistical machine translation. We then introduce NMT in more detail, including the basic framework and the current dominant framework, Transformer, as well as multilingual translation models to deal with the data sparseness problem. In addition, we introduce cutting-edge simultaneous translation methods that achieve a balance between translation quality and latency. We then describe various products and applications of machine translation. At the end of this article, we briefly discuss challenges and future research directions in this field.

Offensive communications have invaded social media content. One of the most effective solutions to cope with this problem is using computational techniques to discriminate offensive content. Moreover, social media users are from linguistically different communities. This study aims to tackle the Multilingual Offensive Language Detection (MOLD) task using transfer learning models and the fine-tuning phase. We propose an effective approach based on the Bidirectional Encoder Representations from Transformers (BERT) that has shown great potential in capturing the semantics and contextual information within texts. The proposed system consists of several stages: (1) Preprocessing, (2) Text representation using BERT models, and (3) Classification into two categories: Offensive and non-offensive. To handle multilingualism, we explore different techniques such as the joint-multilingual and translation-based ones. The first consists in developing one classification system for different languages, and the second involves the translation phase to transform all texts into one universal language then classify them. We conduct several experiments on a bilingual dataset extracted from the Semi-supervised Offensive Language Identification Dataset (SOLID). The experimental findings show that the translation-based method in conjunction with Arabic BERT (AraBERT) achieves over 93% and 91% in terms of F1-score and accuracy, respectively.

Partial discharge (PD) online monitoring is an essential part of transformer insulation condition based maintenance and prognostics and health management (CBM/PHM) in industrial power system. In order to realize online localization of multiple PD sources conveniently, an approach that uses single-channel mixed acoustic emission (AE) signal is proposed in this paper. First, the combined method based on the singular spectrum analysis and the independent component analysis is used to calculate the number of PD sources and realize the separation of mixed AE signals. Next, a difference selection algorithm is proposed to solve the problem that the inherent disorder of signal blind source separation makes the time difference of arrival technology difficult to localize the multiple PD sources directly. Then, the simulation signals and measured signals have verified that the signal separation effect of the proposed single-channel-based method is no less than that of the multi-channel-based method. Finally, the proposed approach is applied to a transformer. The disordered separated signals were successfully grouped to realize multi-source PD localization.

Transformer-based language models have become the de-facto standard model for various NLP applications given the superior algorithmic performances. Processing a transformer-based language model on a conventional accelerator induces the memory wall problem, and the ReRAM-based accelerator is a promising solution to this problem. However, due to the characteristics of the self-attention mechanism and the ReRAM-based accelerator, the pipeline hazard arises when processing the transformer-based language model on the ReRAM-based accelerator. This hazard issue greatly increases the overall execution time. In this paper, we propose a framework to resolve the hazard issue. Firstly, we propose the concept of window self-attention to reduce the attention computation scope by analyzing the properties of the self-attention mechanism. After that, we present a window-size search algorithm, which finds an optimal window size set according to the target application/algorithmic performance. We also suggest a hardware design that exploits the advantages of the proposed algorithm optimization on the general ReRAM-based accelerator. The proposed work successfully alleviates the hazard issue while maintaining the algorithmic performance, leading to a 5.8× speedup over the provisioned baseline. It also delivers up to 39.2×/643.2× speedup/higher energy efficiency over GPU, respectively.

The optimization of Litz-wire gapped-core high-frequency transformer (LGHT) for LLC-converters is a challenging task owing to application requirements and complex relationship between multiple design-objectives and constraints. This necessitates the use of multi-objective optimization-techniques to obtain trade-off design-solutions; however, these are cumbersome methods that require many iterations and resources. Therefore, this paper proposes an improved and efficient Pareto-optimization method for LGHT-design. It combines multi-objective whale-optimization algorithm (MOWOA) and analytical core-geometry factor model (OKGM)-based core and winding selection method while keeping in consideration the LLC-converter requirements. The proposed method utilizes benefits of both method-types; it obtains the best LGHT-design in shortest-time and minimum-iterations with optimized volume and losses, integrated magnetizing-inductance, and satisfied thermal-limit. The method is made resource-efficient by carefully selecting independent and necessary LGHT design-parameters as optimization variables keeping in view interdependency among the parameters and LLC-converter performance; these include peak-flux-density, LGHT design-frequency, core-window utilization-factor, core-material, and core-shape. For best design-selection, supplementary decision functions (total-loss and loss-&-volume-product) are used together with the objective functions (core-loss, winding-loss, and total-volume) to enhance the optimal Pareto-solutions diversity and to assist designers in decision-making. The method is proven via comparison with existing benchmark-methods and validated through performance demonstration of LHGT-prototype in 400VDC-to-12VDC LLC-converter.

The problem of session-aware recommendation aims to predict users' next click based on their current session and historical sessions. Existing session-aware recommendation methods have defects in capturing complex item transition relationships. Other than that, most of them fail to explicitly distinguish the effects of different historical sessions on the current session. To this end, we propose a novel method, named Personalized Graph Neural Networks with Attention Mechanism (A-PGNN) for brevity. A-PGNN mainly consists of two components: one is Personalized Graph Neural Network (PGNN), which is used to extract the personalized structural information in each user behavior graph, compared with the traditional Graph Neural Network (GNN) model, which considers the role of the user when the node embeddding is updated. The other is Dot-Product Attention mechanism, which draws on the Transformer net to explicitly model the effect of historical sessions on the current session. Extensive experiments conducted on two real-world data sets show that A-PGNN evidently outperforms the state-of-the-art personalized session-aware recommendation methods., Comment: 12 pages

Couplet is a unique art form in Chinese traditional culture. The development of deep neural network (DNN) technology makes it possible for computers to automatically generate couplets. Especially, Transformer is a DNN-based ``Encoder-Decoder'' framework, and widely used in natural language processing (NLP). However, the existed Transformer mechanism cannot fully exploit the essential linguistic knowledge in Chinese, including the special format and requirements of Chinese couplets. Therefore, this article adapts the Transformer mechanism to generate meaningful Chinese couplets. Specifically, the contributions of our work are threefold. First, considering the fact that the words in the corresponding positions of the antecedent clause and the subsequent clause in a Chinese couplet always have same part-of-speech (pos, i.e., word class), pos information is intentionally added into the Transformer to improve the accuracy of the conceived couplet. Second, to deal with the large number of unregistered and low-frequency words in Chinese couplet, a specific unregistered/low-frequency word processing mechanism (UWP) is designed and combined with the Transformer model. Third, to further improve the coherence of couplets, we incorporate the polish mechanisms (PMs) into Transformer model. In terms of three evaluation criteria including bilingual evaluation understudy (BLEU), perplexity, and human evaluation, the experimental results demonstrate the effectiveness of our designed Chinese couplet generation system.

Frequency response analysis (FRA) suffers from the interpretation of results despite its potential ability to detect faults related to the power transformer windings. This paper presents a technique for interpreting frequency responses, which is based on image processing and deep learning method and called Graph Convolutional Neural Network (CNN). The proposed procedure transfers frequency responses into 2D images through a visualization technique. The gained images are aggregated into a dataset to be used as the CNN input. The proposed technique is applied on frequency responses of two different winding models with Short Circuit (SC) faults. The SC faults with different intensities are applied on different sections of a simulated ladder model winding and a 20kV winding of a 1.6MVA distribution transformer. After determining the frequency response for each faulty case and applying the visualization technique, the precise locating of SC faults is performed by the CNN. Then, the results are analyzed by performance evaluation metrics. At this stage, the high performance of the CNN in the use of 2D images instead of the conventional method is observed. Finally, by testing the high impedance SC faults in different sections of the simulated winding model and applying the suggested method step by step, early detection of the SC fault is also performed in this study. It should be noted that the suggested technique, in addition to its accuracy and high detection speed, can be considered as an important step in automatic interpretation of frequency responses for online monitoring of transformers

Sentiment analysis uses a series of automated cognitive methods to determine the author's or speaker's attitudes toward an expressed object or text's overall emotional tendencies. In recent years, the growing scale of opinionated text from social networks has brought significant challenges to humans' sentimental tendency mining. The pretrained language model designed to learn contextual representation achieves better performance than traditional learning word vectors. However, the existing two basic approaches for applying pretrained language models to downstream tasks, feature-based and fine-tuning methods, are usually considered separately. What is more, different sentiment analysis tasks cannot be handled by the single task-specific contextual representation. In light of these pros and cons, we strive to propose a broad multitask transformer network (BMT-Net) to address these problems. BMT-Net takes advantage of both feature-based and fine-tuning methods. It was designed to explore the high-level information of robust and contextual representation. Primarily, our proposed structure can make the learned representations universal across tasks via multitask transformers. In addition, BMT-Net can roundly learn the robust contextual representation utilized by the broad learning system due to its powerful capacity to search for suitable features in deep and broad ways. The experiments were conducted on two popular datasets of binary Stanford Sentiment Treebank (SST-2) and SemEval Sentiment Analysis in Twitter (Twitter). Compared with other state-of-the-art methods, the improved representation with both deep and broad ways is shown to achieve a better F1 -score of 0.778 in Twitter and accuracy of 94.0% in the SST-2 dataset, respectively. These experimental results demonstrate the abilities of recognition in sentiment analysis and highlight the significance of previously overlooked design decisions about searching contextual features in deep and broad spaces.

Understanding human language is one of the key themes of artificial intelligence. For language representation, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy texts and getting rid of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanisms for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. The proposed SG-Net is applied to typical Transformer encoders. Extensive experiments on popular benchmark tasks, including machine reading comprehension, natural language inference, and neural machine translation show the effectiveness of the proposed SG-Net design., The early version accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). Journal extension of arXiv:1908.05147 (AAAI 2020)

In the era of social networks, the rapid growth of data mining in information retrieval and natural language processing makes automatic text summarization necessary. Currently, pretrained word embedding and sequence to sequence models can be effectively adapted in social network summarization to extract significant information with strong encoding capability. However, how to tackle the long text dependence and utilize the latent topic mapping has become an increasingly crucial challenge for these models. In this article, we propose a topic-aware extractive and abstractive summarization model named T-BERTSum, based on Bidirectional Encoder Representations from Transformers (BERTs). This is an improvement over previous models, in which the proposed approach can simultaneously infer topics and generate summarization from social texts. First, the encoded latent topic representation, through the neural topic model (NTM), is matched with the embedded representation of BERT, to guide the generation with the topic. Second, the long-term dependencies are learned through the transformer network to jointly explore topic inference and text summarization in an end-to-end manner. Third, the long short-term memory (LSTM) network layers are stacked on the extractive model to capture sequence timing information, and the effective information is further filtered on the abstractive model through a gated network. In addition, a two-stage extractive-abstractive model is constructed to share the information. Compared with the previous work, the proposed model T-BERTSum focuses on pretrained external knowledge and topic mining to capture more accurate contextual representations. Experimental results on the CNN/Daily mail and XSum datasets demonstrate that our proposed model achieves new state-of-the-art results while generating consistent topics compared with the most advanced method.

Ladder network synthesis is very significant for winding fault diagnosis within transformer. To synthesize the network reversely based on the measurable frequency response analysis (FRA) data is the only applicable way. To date the iron core and non-tested winding, which result in the frequency-dependent network components and multiple ladders in some frequency ranges, are usually not considered in the reverse process and thus the synthesis precision is largely limited. This paper proposes to synthesize multiple ladder networks in different frequency regions to handle this problem. Firstly, the sensitivities and frequency-dependent features of network components are analyzed, based on which the topology and components of the networks in diverse frequency ranges are determined afterwards. Further, the network synthesis strategies in different frequency ranges are elaborately designed. Finally, the synthesis method is applied on a distribution transformer successfully, with the fact that the extracted network parameters meet all the constraints and its FRA curve matches well with the measured one. An intact method of multiple-network synthesis for transformer winding in wide frequency range, considering the iron core and non-tested winding, is offered in this paper.

Artificial intelligence shows promising efforts in collaborating the language models with the artificial-intelligence-of-things (AIoT), promoting the edging intelligence on natural language understanding. To adapt to limited computational resources in AIoT, the large language models (e.g., transformer) are compressed into light-weight models, resulting in poor feature representation and unsatisfactory performance on downstream tasks, especially on those low-resource language understanding tasks. To address the above issues, we propose a method named memory-assistant multi-task learning (MAMT), where an auxiliary-memory module is introduced to promote multi-task learning, which serves as a surrogate of target domain representation and performs instance-level weighted multi-task learning. More importantly, our MAMT module is in a plug-and-play fashion. Thus, researchers can plug in it to conduct collaborative training and plug it out for inference without extra computation burdens. Experiments demonstrate that MAMT significantly improves the performance of light-weight models and shows its superiority over the state-of-the-arts on eight GLUE sub-tasks.

We propose a model driven Bayesian deep learning framework for multiple access uplink systems in Multi-user MIMO systems. Utilising tools from Streaming Variational Inference, we combine graphical models with neural networks to enable fast online machine learning. The proposed distributed inference framework is shown to be robust and suitable for online scenario. Our simulations demonstrate the robustness of the proposed solution in online propagation environments and its ability to capture uncertainty.

Owing to the considerable shortage of traditional energy sources, utilization of renewable energy is gaining attention. Thus, the high voltage inverters for solar and wind power generation systems are in huge demand. Modular inverters are more preferred over singular capacity inverters due to their numerous benefits such as relieving of thermal management, fault tolerance, reduced component stress and modularity. This paper proposes a parallel-configured improved Z-source inverter (ZSI) based on transformer. The proposed circuit has increased voltage gain with the combined benefits of paralleling inverters. Also, it offers lower harmonic distortion and lower filter requirements by employing the interleaved pulse-width-modulation (PWM) strategy. Two parallel-connected transformer-based improved ZSI is firstly analyzed in detail, and the similar concept is then applied to N parallel-connected transformer-based improved ZSI. A hardware prototype is implemented in laboratory, and it is tested experimentally for high power applications.

Nowadays, the scale of power grid is remarkably increasing and there is a continuous reinforcement of AC–DC hybrid systems. Thus, the single-phase grounding fault currents on the AC grid-side of rectifier and inverter stations increase significantly, which may exceed the capacity limit of the circuit breakers. The large short-circuit currents would result in problems during equipment selection and threaten the safety of equipment and personnel. To solve such issue, the effect of the converter transformer on the single-phase grounding fault current on the grid-side is analyzed. The mechanism of high short-circuit current in DC near region is revealed. Also, the feasibility of neutral point earthing via small reactor at HVDC converter transformer to suppressing the single-phase grounding fault current is verified. In addition, the principle of selecting the suitable small reactor is summarized. The results show that the neutral point earthing via small reactor at converter transformer can effectively limit the single-phase grounding fault current on the AC-side. However, it would lead to an increase of overvoltage at the neutral point of HVDC transformer. Therefore, neutral grounding by small reactor might be implemented when the insulation level is satisfied.

The strong demand of autonomous driving in the industry has led to vigorous interest in 3D object detection and resulted in many excellent 3D object detection algorithms. However, the vast majority of algorithms only model single-frame data, ignoring the temporal clue in video sequence. In this work, we propose a new transformer, called Temporal-Channel Transformer (TCTR), to model the temporal-channel domain and spatial-wise relationships for video object detecting from Lidar data. As the special design of this transformer, the information encoded in the encoder is different from that in the decoder. The encoder encodes temporal-channel information of multiple frames while the decoder decodes the spatial-wise information for the current frame in a voxel-wise manner. Specifically, the temporal-channel encoder of the transformer is designed to encode the information of different channels and frames by utilizing the correlation among features from different channels and frames. On the other hand, the spatial decoder of the transformer decodes the information for each location of the current frame. Before conducting the object detection with detection head, a gate mechanism is further deployed for re-calibrating the features of current frame, which filters out the object-irrelevant information by repetitively refining the representation of target frame along with the up-sampling process. Experimental results reveal that TCTR achieves the state-of-the-art performance in grid voxel-based 3D object detection on the nuScenes benchmark.

Industrial process data has characteristics of high non-linearity, dynamics and noises. And the soft sensing methods considering timing characteristics based on deep learning are usually faced with gradient vanishing and the difficulty in parallel computing. Therefore, a novel Gated Convolutional neural network-based Transformer is proposed for dynamic soft sensor modeling of industrial processes. The GCT encodes short-term patterns of time series data and filters important features adaptively through an improved gated convolutional neural network (CNN). Then, the multi-head attention mechanism is applied to modeling the correlation between any two moments. Finally, the prediction results are obtained through a linear neural network layer with the highway connection. The experiments in the dynamic soft sensor modeling of polypropylene and purified terephthalic acid industrial processes show that the proposed method achieves state-of-the-art comparing with back propagation neural network, extreme learning machine, long short-term memory (LSTM) and LSTM based on the CNN.

The increasing computational complexity of DNNs achieved unprecedented successes in various areas such as machine vision and natural language processing (NLP), e.g., the recent advanced Transformer has billions of parameters. However, as large-scale DNNs significantly exceed GPU’s physical memory limit, they cannot be trained by conventional methods such as data parallelism. Pipeline parallelism that partitions a large DNN into small subnets and trains them on different GPUs is a plausible solution. Unfortunately, the layer partitioning and memory management in existing pipeline parallel systems are fixed during training, making them easily impeded by out-of-memory errors and the GPU under-utilization. These drawbacks amplify when performing neural architecture search (NAS) such as the evolved Transformer, where different network architectures of Transformer needed to be trained repeatedly. vPipe is the first system that transparently provides dynamic layer partitioning and memory management for pipeline parallelism. vPipe has two unique contributions, including (1) an online algorithm for searching a near-optimal layer partitioning and memory management plan, and (2) a live layer migration protocol for re-balancing the layer distribution across a training pipeline. vPipe improved the training throughput of two notable baselines (Pipedream and GPipe) by 61.4-463.4 percent and 24.8-291.3 percent on various large DNNs and training settings.

Anomaly detection for smart contracts can effectively prevent hidden security risks such as financial fraud, illegal financing, and money laundering. Ethereum is currently the largest platform for smart contracts, and anomaly detection is imminent. However, the data related to smart contracts is huge and contains complex objects and relationships. It is impossible to extract high-order attributes and low efficiency using traditional methods. The key to reduce fraud is extracting features from complex smart contracts and effectively identifying abnormal contracts. Therefore, this paper constructs a Heterogeneous Graph Transformer Networks (S_HGTNs) suitable for smart contract anomaly detection to detect financial fraud on the Ethereum platform. To perform feature representation, this paper first extracts the features to construct a Heterogeneous Information Network (HIN) for smart contract, and uses the relationship matrix obtained from the meta-path learned in the transformer network as the input of the convolution network, and finally uses the node embedding for classification tasks. The classification results show that this model performs better than the traditional model and the standard deviation is small, which proves the effectiveness and stability of the model.

This paper presents a novel health management method of the dry-type transformer to diagnose the early unhealthy behavior of the transformer and evaluate the transformer's health conditions by health score. The health condition diagnosis implemented by a proposed DW-FB-BLS(Dynamic-weighting-Feed-Back-BLS) method, which helps to determine the BLS network structure effectively, and adjusts the weight of features in the online application to avoid reduction of accuracy caused by concept drift. Then, a rational score rule is set to evaluate the health condition of the dry-type transformer by health score, which allows intuitive presentation and preservation of the transformer's health condition over a long period. Finally, the effectiveness and validity of the proposed method are verified based on the real field data of the dry-type transformer. Satisfactory results for unhealthy behavior diagnosis and health evaluation were obtained, it shows that health management of this work can reflect the real health condition appropriately.

Owing to its superior performance, the Transformer model, based on the 'Encoder- Decoder' paradigm, has become the mainstream model in natural language processing. However, bioinformatics has embraced machine learning and has led to remarkable progress in drug design and protein property prediction. Cell-penetrating peptides (CPPs) are a type of permeable protein that is a convenient 'postman' in drug penetration tasks. However, only a few CPPs have been discovered, limiting their practical applications in drug permeability. CPPs have led to a new approach that enables the uptake of only macromolecules into cells (i.e., without other potentially harmful materials found in the drug). Most previous studies have utilized trivial machine learning techniques and hand-crafted features to construct a simple classifier. CPPFormer was constructed by implementing the attention structure of the Transformer, rebuilding the network based on the characteristics of CPPs according to their short length, and using an automatic feature extractor with a few manually engineered features to co-direct the predicted results. Compared to all previous methods and other classic text classification models, the empirical results show that our proposed deep model-based method achieves the best performance, with an accuracy of 92.16% in the CPP924 dataset, and passes various index tests.

This article presents the modified Preisach model of hysteresis for a 3-phase medium frequency transformer in a 100 kW dual active bridge converter. The transformer magnetic core is assembled out of ferrite I-cores, which results in multiple parasitic air gaps. For this transformer, the hysteresis loops were measured and parameters of the Preisach model were determined. The Preisach distribution function is approximated with a two-dimensional Gauss function series and the feedback function is a 3rd-degree polynomial. The optimized identification of Preisach distribution function parameters was prepared. Two sets of parameters were determined based on the analysis of major and minor hysteresis loop. The developed model is used to analyze the transformer core power loss. A new set of Steinmetz equation parameters for multi air gap ferrite core MFT is proposed.

This letter presents a millimeter-wave silicon-germanium (SiGe) distributed attenuator which uses a transformer-based attenuation core. The nonlinearities of the SiGe heterojunction bipolar transistor (HBT) are managed using an RC-load, which linearizes the device impedance versus bias. In addition, a dual-resonance transformer is used to support broadband operation, while providing control over the attenuator insertion phase. Two attenuators, one using transmission line-based impedance transformers, and one using transformer-based impedance transformers, were designed and fabricated in a 180-nm SiGe bipolar and CMOS (BiCMOS) technology platform. The attenuators achieve competitive performance in terms of bandwidth and phase error, but with low insertion loss and large attenuation range.

Drusen is considered as the landmark for diagnosis of AMD and important risk factor for the development of AMD. Therefore, accurate segmentation of drusen in retinal OCT images is crucial for early diagnosis of AMD. However, drusen segmentation in retinal OCT images is still very challenging due to the large variations in size and shape of drusen, blurred boundaries, and speckle noise interference. Moreover, the lack of OCT dataset with pixel-level annotation is also a vital factor hindering the improvement of drusen segmentation accuracy. To solve these problems, a novel multi-scale transformer global attention network (MsTGANet) is proposed for drusen segmentation in retinal OCT images. In MsTGANet, which is based on U-Shape architecture, a novel multi-scale transformer non-local (MsTNL) module is designed and inserted into the top of encoder path, aiming at capturing multi-scale non-local features with long-range dependencies from different layers of encoder. Meanwhile, a novel multi-semantic global channel and spatial joint attention module (MsGCS) between encoder and decoder is proposed to guide the model to fuse different semantic features, thereby improving the model's ability to learn multi-semantic global contextual information. Furthermore, to alleviate the shortage of labeled data, we propose a novel semi-supervised version of MsTGANet (Semi-MsTGANet) based on pseudo-labeled data augmentation strategy, which can leverage a large amount of unlabeled data to further improve the segmentation performance. Finally, comprehensive experiments are conducted to evaluate the performance of the proposed MsTGANet and Semi-MsTGANet. The experimental results show that our proposed methods achieve better segmentation accuracy than other state-of-the-art CNN-based methods.

Transformers are widely used in natural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the natural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature., 22 pages, 14 figures, 3 appendices

The high‐voltage isolated energy supply transformer (HIET) is the key component of the HVDC breaker, which is used for energy transmission and the potential isolation. There is a lack of research results that can meet engineering applications at home and abroad. The design of a 500 kV HIET with dry‐type insulation is proposed, using 10 cascade connected 50 kV sub‐transformers (STs), which uses silicon rubber suitable for DC voltage as the main insulation material. High‐voltage (HV) winding of ST adopts double‐shielding structure of coil semi‐conductive shielding and inner semi‐conductive shielding to improve electric field distribution. The simulation analysis of an electric field at different insulation thicknesses and temperatures shows that the electric field decreases with an increase in thickness of insulation. When the thickness is greater than 25 mm, the electric field reduction effect is significantly weakened. The electric field is closely related to temperature, when at 20°C, the electric field between double‐shielding is relatively uniform. When at 70°C, the inner shielding electric field is much higher than coil shielding, and undergoes obvious reversal. In addition, the 500 kV HIET's electric field has been analysed. The 50 0kV HIET prototype has been manufactured and passed the type test, and successfully applied to the 500 kV Zhangbei DC grid project.

Distant supervised relation extraction is an efficient strategy of finding relational facts from unstructured text without labeled training data. A recent paradigm to develop relation extractors is using pre-trained Transformer language models to produce high-quality sentence representations. However, due to the original Transformer is weak at capturing local dependencies and phrasal structures, existing Transformer-based methods cannot identify various relational patterns in sentences. To address this issue, we propose a novel distant supervised relation extraction model, which employs a specific-designed pattern-aware self-attention network to automatically discover relational patterns for pre-trained Transformers in an end-to-end manner. Specifically, the proposed method assumes that the correlation between two adjacent tokens reflects the probability that they belong to the same pattern. Based on this assumption, a novel self-attention network is designed to generate the probability distribution of all patterns in a sentence. Then, the probability distribution is applied as a constraint in the first Transformer layer to encourage its attention heads to follow the relational pattern structures. As a result, fine-grained pattern information is enhanced in the pre-trained Transformer without losing global dependencies. Extensive experimental results on two popular benchmark datasets demonstrate that our model performs better than the state-of-the-art baselines.

This paper investigates the cause of temporary overvoltage in PV system and different ways to mitigate them. Temporary overvoltage is an undesired phenomenon in distributed generation system which needs to be eliminated using different transformer configuration and grounding banks. This paper illustrates the principles of transformer configuration for mitigation of temporary overvoltage in PV system.

The established approach to unsupervised protein contact prediction estimates coevolving positions using undirected graphical models. This approach trains a Potts model on a Multiple Sequence Alignment. Increasingly large Transformers are being pretrained on unlabeled, unaligned protein sequence databases and showing competitive performance on protein contact prediction. We argue that attention is a principled model of protein interactions, grounded in real properties of protein family data. We introduce an energy-based attention layer, factored attention, which, in a certain limit, recovers a Potts model, and use it to contrast Potts and Transformers. We show that the Transformer leverages hierarchical signal in protein family databases not captured by single-layer models. This raises the exciting possibility for the development of powerful structured models of protein family databases.

The success of deep learning methods led to significant breakthroughs in 3-D point cloud processing tasks with applications in remote sensing. Existing methods utilize convolutions that have some limitations, as they assume a uniform input distribution and cannot learn long-range dependences. Recent works have shown that adding attention in conjunction with these methods improves performance. This raises a question: can attention layers completely replace convolutions? This letter proposes a fully attentional model--Point Transformer (PT) for deriving a rich point cloud representation. The model's shape classification and retrieval performance are evaluated on a large-scale urban data set--RoofN3D and a standard benchmark data set ModelNet40. Extensive experiments are conducted to test the model's robustness to unseen point corruptions for analyzing its effectiveness on real data sets. The proposed method outperforms other state-of-the-art models in the RoofN3D data set, gives competitive results in the ModelNet40 benchmark, and shows high robustness to various unseen point corruptions. Furthermore, the model is highly memory and space-efficient when compared to other methods.

Although video summarization has achieved tremendous success benefiting from Recurrent Neural Networks (RNN), RNN-based methods neglect the global dependencies and multi-hop relationships among video frames, which limits the performance. Transformer is an effective model to deal with this problem, and surpasses RNN-based methods in several sequence modeling tasks, such as machine translation, video captioning, etc. Motivated by the great success of transformer and the natural structure of video (frame-shot-video), a hierarchical transformer is developed for video summarization, which can capture the dependencies among frame and shots, and summarize the video by exploiting the scene information formed by shots. Furthermore, we argue that both the audio and visual information are essential for the video summarization task. To integrate the two kinds of information, they are encoded in a two-stream scheme, and a multimodal fusion mechanism is developed based on the hierarchical transformer. In this paper, the proposed method is denoted as Hierarchical Multimodal Transformer (HMT). Practically, extensive experiments show that HMT achieves (F-measure: 0.441, Kendall’s τ : 0.079, Spearman’s ρ : 0.080) and (F-measure: 0.601, Kendall’s τ : 0.096, Spearman’s ρ : 0.107) on SumMe and TVsum, respectively. It surpasses most of the traditional, RNN-based and attention-based video summarization methods.