Search

Textual logical reasoning, especially question-answering (QA) tasks with logical reasoning, requires awareness of particular logical structures. The passage-level logical relations represent entailment or contradiction between propositional units (e.g., a concluding sentence). However, such structures are unexplored as current QA systems focus on entity-based relations. In this work, we propose logic structural-constraint modeling to solve the logical reasoning QA and introduce discourse-aware graph networks (DAGNs). The networks first construct logic graphs leveraging in-line discourse connectives and generic logic theories, then learn logic representations by end-to-end evolving the logic relations with an edge-reasoning mechanism and updating the graph features. This pipeline is applied to a general encoder, whose fundamental features are joined with the high-level logic features for answer prediction. Experiments on three textual logical reasoning datasets demonstrate the reasonability of the logical structures built in DAGNs and the effectiveness of the learned logic features. Moreover, zero-shot transfer results show the features' generality to unseen logical texts.

Medical reports have significant clinical value to radiologists and specialists, especially during a pandemic like COVID. However, beyond the common difficulties faced in the natural image captioning, medical report generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure. Secondly, there are many similar sentences used in each medical report to describe the normal regions of the image, which causes serious data bias. This deviation is likely to teach models to generate these inessential sentences on a regular basis. To address these problems, we propose an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) to mimic radiologists’ working patterns. Specifically, the auxiliary patches are explored to expand the widely used visual patch features before fed to the Transformer encoder, while the external linguistic signals help the decoder better master prior knowledge during the pre-training process. Our approach performs well on common benchmarks, including CX-CHR, IU X-Ray, and COVID-19 CT Report dataset (COV-CTR), demonstrating combining auxiliary signals with transformer architecture can bring a significant improvement in terms of medical report generation. The experimental results confirm that auxiliary signals driven Transformer-based models are with solid capabilities to outperform previous approaches on both medical terminology classification and paragraph generation metrics.

The whale optimization algorithm (WOA) is a metaheuristic algorithm based on swarm intelligence and it mimics the hunting behavior of whales. It has the imperfection of premature convergence into local optima. In order to overcome this disadvantage, a multioperator WOA (MOWOA) is proposed. Four main strategies are introduced to the MOWOA to heighten the search capacity of WOA. The strategies include nonlinear adaptive parameter design, an exploration mechanism of honey badger, Cauchy factor strategy, and greedy strategy. This paper tests the versatility of MOWOA with three different types of benchmark functions, and a kind of seismic inversion problem are trialed run. From the experimental results, the performance of MOWOA outperforms the compared algorithms in global optimization.

Beyond generating long and topic-coherent paragraphs in traditional captioning tasks, the medical image report composition task poses more task-oriented challenges by requiring both the highly-accurate medical term diagnosis and multiple heterogeneous forms of information including impression and findings. Current methods often generate the most common sentences due to dataset bias for individual case, regardless of whether the sentences properly capture key entities and relationships. Such limitations severely hinder their applicability and generalization capability in medical report composition where the most critical sentences lie in the descriptions of abnormal diseases that are relatively rare. Moreover, some medical terms appearing in one report are often entangled with each other and co-occurred, e.g. symptoms associated with a specific disease. To enforce the semantic consistency of medical terms to be incorporated into the final reports and encourage the sentence generation for rare abnormal descriptions, we propose a novel framework that unifies template retrieval and sentence generation to handle both common and rare abnormality while ensuring the semantic-coherency among the detected medical terms. Specifically, our approach exploits hybrid-knowledge co-reasoning: i) explicit relationships among all abnormal medical terms to induce the visual attention learning and topic representation encoding for better topic-oriented symptoms descriptions; ii) adaptive generation mode that changes between the template retrieval and sentence generation according to a contextual topic encoder. Experimental results on two medical report benchmarks demonstrate the superiority of the proposed framework in terms of both human and metrics evaluation., To appear in IEEE Transactions on Cybernetics 2021. We attempt to resolve the challenging medical report composition task by i) enforcing the semantic consistency of medical terms to be incorporated into the final reports; and ii) encouraging the sentence generation for rare abnormal descriptions

The whale optimization algorithm (WOA) is inspired by humpback whale social behavior and WOA is a popular swarm intelligence algorithm. Yet, the WOA does have certain flaws, such as a restricted global search capability and an inconsistent convergence speed, and when dealing with complex optimization problems, WOA is easy to fall into local optimum. To address the WOA’s shortcomings, a modified whale optimization algorithm with convergence and exploitability enhancement called MWOA-CEE is proposed. Three operators, opposition-based learning, exponentially decreasing function, and elite-guided Cauchy mutation, are integrated into the WOA to enhance the exploration performance and avoid local optimum. First, the opposition-based learning used the opposition-based concept to generate the opposite position of the candidate solution, which can enhance the convergence ability and quality of the solution. Then, the exponentially decreasing function balances the exploitation and exploration phases and boosts the exploitation capability. In the end, the elite-guided Cauchy mutation generated a wide disturbance near the current candidate solution, which improves solutions diversity and increases the algorithm’s global search capability. MWOA-CEE is validated on 19 optimization benchmark functions and 14 CEC2014 benchmark functions and used to identify IIR systems. It is compared to seven of the commonly used algorithms. The results prove that MWOA-CEE outperforms the other algorithms in terms of final solution quality and convergence rate.

As an important part of the soft power of urban culture, the importance of city image has received widespread attention from academic circles and society. The study of city image communication has become one of the important research directions in the field of communication in recent years. This research mainly discusses the application of 5G virtual reality in the design and dissemination of a contemporary urban image system under the background of big data. This research designs and develops a city image system that models and simulates virtual cities in a three-dimensional scene. The system consists of several different modules, including a data acquisition module, scene construction, and roaming module. The most important data resource formats used by this system are the DXF format and 3DS format, which can be created and acquired using digital stereo images in the data acquisition module. The data import and export module can convert the models in DXF format and 3DS format into Openfligt format models. The model editing and reconstruction module can visually edit and reconstruct the converted model, so that the space and attribute information of the recreated model is stored in the data storage and management system. Finally, a scene roaming module is created, and models are selected from the database to create a 3D city scene and provide interactive roaming information extraction. After the implementation of virtual reality technology, the per capita awareness of urban humanities reached 63.2%. The test results show that because the system integrates the respective advantages of the geographic information system, virtual reality, and database technology, it can better realize the modeling and simulation of digital cities under the conditions of three-dimensional visualization, which is conducive to the spread of urban image.

Graph-level representations are critical in various real-world applications, such as predicting the properties of molecules. However, in practice, precise graph annotations are generally very expensive and time-consuming. To address this issue, graph contrastive learning constructs an instance discrimination task, which pulls together positive pairs (augmentation pairs of the same graph) and pushes away negative pairs (augmentation pairs of different graphs) for unsupervised representation learning. However, since for a query, its negatives are uniformly sampled from all graphs, existing methods suffer from the critical sampling bias issue, i.e., the negatives likely having the same semantic structure with the query, leading to performance degradation. To mitigate this sampling bias issue, in this article, we propose a prototypical graph contrastive learning (PGCL) approach. Specifically, PGCL models the underlying semantic structure of the graph data via clustering semantically similar graphs into the same group and simultaneously encourages the clustering consistency for different augmentations of the same graph. Then, given a query, it performs negative sampling via drawing the graphs from those clusters that differ from the cluster of query, which ensures the semantic difference between query and its negative samples. Moreover, for a query, PGCL further reweights its negative samples based on the distance between their prototypes (cluster centroids) and the query prototype such that those negatives having moderate prototype distance enjoy relatively large weights. This reweighting strategy is proven to be more effective than uniform sampling. Experimental results on various graph benchmarks testify the advantages of our PGCL over state-of-the-art methods. The code is publicly available at https://github.com/ha-lins/PGCL.

The success of deep learning has brought breakthroughs in many fields. However, the increased performance of deep learning models is often accompanied by an increase in their depth and width, which conflicts with the storage, energy consumption, and computational power of edge devices. Knowledge distillation, as an effective model compression method, can transfer knowledge from complex teacher models to student models. Self-distillation is a special type of knowledge distillation, which does not to require a pre-trained teacher model. However, existing self-distillation methods rarely consider how to effectively use the early features of the model. Furthermore, most self-distillation methods use features from the deepest layers of the network to guide the training of the branches of the network, which we find is not the optimal choice. In this paper, we found that the feature maps obtained by early feature fusion do not serve as a good teacher to guide their own training. Based on this, we propose a selective feature fusion module and further obtain a new self-distillation method, knowledge fusion distillation. Extensive experiments on three datasets have demonstrated that our method has comparable performance to state-of-the-art distillation methods. In addition, the performance of the network can be further enhanced when fused features are integrated into the network.

Purpose Studies on titanium implants have shown that the mechanical properties of the parts are affected by the microstructure characteristic derived from the manufacturing process. The properties of different orientations of specimens under the same process parameters will be different, which should be considered in the application of bone implants. This paper aims to understand the influence of microstructure on micro-hardness, wear and corrosion resistance in different orientations. Design/methodology/approach The authors manufactured titanium parts and carried out micro-hardness, wear tests and electrochemical corrosion of different orientations under the same process conditions. Then, finally studied the evolution mechanism of the microstructure in different orientations and its influence mechanism on wear and corrosion mechanism. Findings The melting method makes the grains on the surface in XY orientation finer. The wear mechanism of XY orientation is abrasive wear, that of XZ and YZ orientations are adhesive wear. During corrosion, XY orientation forms a stable passivation film earlier. Compared with XZ and YZ orientations, XY orientation has higher micro-hardness, better wear and corrosion resistance. Originality/value In this paper, the microstructure, wear and corrosion resistance of selective laser melted parts were discussed and the differences in different orientations under the same experimental conditions were discussed. The evolution mechanism of the microstructure in different orientations and its influence mechanism on wear mechanism and corrosion mechanism was studied. The mechanical anisotropy of selective laser melted components was discussed.

Multi-energy industrial parks are required to render a huge variety of services in an eco-friendly, secure, reliable, and affordable way. The industrial energy park is a separate area consisting of multiple distributed generations, energy storage systems, etc., which supply local gas, heating, and electrical consumers. Meanwhile, the integration of power-to-X technologies such as power-to-gas and power-to-heat, which convert the electricity into other forms of energies while facilitating the integration of renewable energy in the industrial park, can enhance the flexibility and efficiency of energy supply. Therefore, this paper proposes novel robust energy management of multi-energy industrial parks integrated with wind power resources, cogeneration units, power-to-X technologies, and demand response programs to total operation cost minimization. The industrial park can simultaneously participate in a multi-energy market, including power, thermal, and gas markets, to meet local heating, gas, and electrical load. The robust optimization framework is extended to address the power price uncertainty and manage the conservatism level of the operator against price variability. The proposed model is examined on the industrial park test system, and numerical results will be presented for the different cases. Under the robust energy management, the total operation cost of the multi-energy industrial park reduces up to 53 %.

Circular oscillating laser was used to deposit GH3536 nickel-based superalloy, GH3536-16 wt.% coarse WC and GH3536-16 wt.% fine WC composite. The microstructure characteristics and evolution, phase composition and properties of the three materials fabricated by circular oscillating laser melting deposition were studied. In the mode of circular oscillating laser, equiaxed and columnar crystal structures are formed in the middle and edge of the cladding track of the three materials due to its temperature gradient distribution and the laser stirring effect on molten pool. The main phases of the composite coatings are γ-Ni solid solution, WC, W2C and M6C. WC particles uniformly distributed in GH3536 matrix have the effects of fine grain strengthening and dispersion strengthening. Compared with GH3536, the average microhardness of the composite material is increased by 59.70% (coarse WC) and 74.66% (fine WC) respectively, and the wear rate is reduced by 84.87% (coarse WC) and 89.17% (fine WC) respectively. The wear mechanism changes from adhesive wear to abrasive wear after adding WC. The corrosion current density of the composite coating is 44.9% (coarse WC) and 26.5% (fine WC) lower than that of GH3536, respectively. WC particles have physical shielding and solid solution strengthening effects, and improve the corrosion resistance of GH3536 coating.

Visual dialog is a challenging vision-language task, which requires an agent to answer a series of questions about an image. To answer correctly, the agent should explore question-relevant clues from heterogeneous information, such as resolve visual coreference using dialog history, or locate the referred object in the image. Previous methods usually capture question-relevant factors from heterogeneous information in a sequential manner, that is, first enrich the question context with relevant dialog history, and then use the enriched question to perform visual grounding. However, this sequential way may bring unexpected information mix, leading to wrong answers. In this paper, we propose Heterogeneous Excitation-and-Squeeze Network (HESNet), which can handle with heterogeneous information in a parallel and adaptive way. HESNet contains two key modules: Heterogeneous Excitation Module (HE-Module) and Heterogeneous Squeeze Module (HS-Module). HE-Module excavates question-related clues in heterogeneous information, namely image, dialog history, and candidate answer parallelly by conducting bi-directional excitation operations. Then, the HS-Module squeezes the enriched question representation and multiple enhanced heterogeneous representations adaptively for final answer prediction. Experimental results on VisDial v0.9 and v1.0 datasets show the superiority of the proposed approach over the state-of-the-art methods. The visualization analysis further shows that the proposed HESNet alleviates the information mix by collecting the heterogeneous clues parallelly and adaptively.

Non-parallel text style transfer has attracted increasing research interests in recent years. Despite successes in transferring the style based on the encoder-decoder framework, current approaches still lack the ability to preserve the content and even logic of original sentences, mainly due to the large unconstrained model space or too simplified assumptions on latent embedding space. Since language itself is an intelligent product of humans with certain grammars and has a limited rule-based model space by its nature, relieving this problem requires reconciling the model capacity of deep neural networks with the intrinsic model constraints from human linguistic rules. To this end, we propose a method called Graph Transformer–based Auto-Encoder, which models a sentence as a linguistic graph and performs feature extraction and style transfer at the graph level, to maximally retain the content and the linguistic structure of original sentences. Quantitative experiment results on three non-parallel text style transfer tasks show that our model outperforms state-of-the-art methods in content preservation, while achieving comparable performance on transfer accuracy and sentence naturalness.

Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. In this paper, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, that can generalized well on multiple dimensions in both CNNs and transformers (e.g. kernel size, embedding dimension, number of heads, etc.). Instead of adaptively selecting important weight elements in a sparse way, we pre-define dense weight slices with different importance level by nested residual learning. During inference, weights are progressively sliced beginning with the most important elements to less important ones to achieve different model capacity for inputs with diverse difficulty levels. Based on this conception, we present DS-CNN++ and DS-ViT++, by carefully designing the double headed dynamic gate and the overall network architecture. We further propose dynamic idle slicing to address the drastic reduction of embedding dimension in DS-ViT++. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme. In Stage I, in-place bootstrapping (IB) and multi-view consistency (MvCo) are proposed to stablize and improve the training of DS-CNN++ and DS-ViT++ supernet, respectively. In Stage II, sandwich gate sparsification (SGS) is proposed to assist the gate training. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our methods consistently outperform the state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, we achieves 2-4× computation reduction and up to 61.5% real-world acceleration on MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops on ImageNet. Code release: https://github.com/changlin31/DS-Net.

The annular coaxial nozzle has good convergence and is widely used in laser melting deposition (LMD). In this paper, a new-type annular coaxial nozzle is developed, which can transform four-channel powder flow into annular distribution through the internal channel to obtain a more uniform distribution of powder in the circumferential direction. A three-dimensional computational fluid dynamics (CFD) numerical model is established. The dynamic behavior of Fe-based powder is described by discrete phase model (DPM). The influence of gas on powder particles is described by two-way turbulence coupling model. The convergence and transport characteristics of the nozzle are revealed. The accuracy of CFD model is verified by powder feeding experiment, and the deposition performance of powder nozzle is verified by single-track deposition experiment. The results show that the nozzle can provide stable powder feeding and excellent convergence characteristics. The powder flow is converged in a “Y” shape on the longitudinal section. The higher the powder concentration at the focal plane of the nozzle, the smaller the aspect ratio of single-track coating. The powder utilization rate of the nozzle can reach 76.97%.

When answering a question, people often draw upon their rich world knowledge in addition to the particular context. While recent works retrieve supporting facts/evidence from commonsense knowledge bases to supply additional information to each question, there is still ample opportunity to advance it on the quality of the evidence. It is crucial since the quality of the evidence is the key to answering common- sense questions, and even determines the upper bound on the QA systems’ performance. In this paper, we propose a recursive erasure memory network (REM-Net) to cope with the quality improvement of evidence. To address this, REM-Net is equipped with a module to refine the evidence by recursively erasing the low-quality evidence that does not explain the question answering. Besides, instead of retrieving evidence from existing knowledge bases, REM-Net leverages a pre-trained generative model to generate candidate evidence customized for the question. We conduct experiments on two commonsense question answering datasets, WIQA and CosmosQA. The results demonstrate the performance of REM- Net and show that the refined evidence is explainable.

A new approach is presented in this paper to link the integrated area of the spectral curve with the molten pool temperature, the grain size and molten pool plasma during laser additive manufacturing. A fiber laser was used to deposit the CoCrMo powder on the Q235 steel substrate. The spectra generated during the additive manufacturing process were collected with an optical spectrometer attached to the laser head. According to the thermal radiation theory and the rapid solidification theory, spectral information was applied to calculate the molten pool temperature and the grain size. The relationship between the molten pool temperature and the integrated area of the spectral curve, as well as the relationship between the grain size and the integrated area of the spectral curve was established, respectively. Meanwhile, the electron temperature and electron density of the molten pool plasma were calculated from two different Cr I spectral lines at 403.292 and 434.429 nm wavelength. The results show that the molten pool temperature and the grain size are closely related to the integrated area of the spectral curve. Furthermore, the electron temperature and electron density also increase with the integrated area of the spectral curve.

Collaborative reasoning for understanding image-question pairs is a very critical but underexplored topic in interpretable visual question answering systems. Although very recent studies have attempted to use explicit compositional processes to assemble multiple subtasks embedded in questions, their models heavily rely on annotations or handcrafted rules to obtain valid reasoning processes, which leads to either heavy workloads or poor performance on compositional reasoning. In this paper, to better align image and language domains in diverse and unrestricted cases, we propose a novel neural network model that performs global reasoning on a dependency tree parsed from the question; thus, our model is called a parse-tree-guided reasoning network (PTGRN). This network consists of three collaborative modules: i) an attention module that exploits the local visual evidence of each word parsed from the question, ii) a gated residual composition module that composes the previously mined evidence, and iii) a parse-tree-guided propagation module that passes the mined evidence along the parse tree. Thus, PTGRN is capable of building an interpretable visual question answering (VQA) system that gradually derives image cues following question-driven parse-tree reasoning. Experiments on relational datasets demonstrate the superiority of PTGRN over current state-of-the-art VQA methods, and the visualization results highlight the explainable capability of our reasoning system., Comment: 14 pages, 10 figures. arXiv admin note: text overlap with arXiv:1804.00105

Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features from the teacher to the student in a one-to-one spatial matching fashion. However, people tend to overlook the fact that, due to the architecture differences, the semantic information on the same spatial location usually vary. This greatly undermines the underlying assumption of the one-to-one distillation approach. To this end, we propose a novel one-to-all spatial matching knowledge distillation approach. Specifically, we allow each pixel of the teacher feature to be distilled to all spatial locations of the student features given its similarity, which is generated from a target-aware transformer. Our approach surpasses the state-of-the-art methods by a significant margin on various computer vision benchmarks, such as ImageNet, Pascal VOC and COCOStuff10k. Code will be released soon., CVPR2022(Oral)

Few-shot class-incremental learning (FSCIL) has two main problems: (1) catastrophically forgetting old classes while feature representations drift into new classes, and (2) over-fitting new classes for the few training samples available. Revealed by our analyses, the problems are caused by feature distribution crumbling, which causes class confusion when continuously embedding few samples to a fixed feature space. In this study, we propose a Dynamic Support Network (DSN), which refers to an adaptively updating network with compressive node expansion to 'support' the feature space. In each training session, DSN tentatively expands network nodes to enlarge feature representation capacity for incremental classes. It then dynamically compresses the expanded network by node self-activation to pursue compact feature representation which alleviates over-fitting. Simultaneously, DSN selectively recalls old class distributions during incremental learning process to support feature distributions and avoid confusion between classes. DSN with compressive node expansion and class distribution recalling provides a systematic solution for the problems of catastrophically forgetting and overfitting. Experiments on CUB, CIFAR-100, and miniImage datasets show that DSN significantly improves upon the baseline approach, achieving new state-of-the-arts. The code is publicly available.

Continual learning is a challenging real-world problem for constructing a mature AI system when data are provided in a streaming fashion. Despite recent progress in continual classification, the researches of continual object detection are impeded by the diverse sizes and numbers of objects in each image. Different from previous works that tune the whole network for all tasks, in this work, we present a simple and flexible framework for continual object detection via pRotOtypical taSk corrElaTion guided gaTing mechAnism (ROSETTA). Concretely, a unified framework is shared by all tasks while task-aware gates are introduced to automatically select sub-models for specific tasks. In this way, various knowledge can be successively memorized by storing their corresponding sub-model weights in this system. To make ROSETTA automatically determine which experience is available and useful, a prototypical task correlation guided Gating Diversity Controller(GDC) is introduced to adaptively adjust the diversity of gates for the new task based on class-specific prototypes. GDC module computes class-to-class correlation matrix to depict the cross-task correlation, and hereby activates more exclusive gates for the new task if a significant domain gap is observed. Comprehensive experiments on COCO-VOC, KITTI-Kitchen, class-incremental detection on VOC and sequential learning of four tasks show that ROSETTA yields state-of-the-art performance on both task-based and class-based continual object detection.

With the rise of telemedicine, the task of developing Dialogue Systems for Medical Diagnosis (DSMD) has received much attention in recent years. Different from early researches that needed to rely on extra human resources and expertise to help construct the system, recent researches focused on how to build DSMD in a purely data-driven manner. However, the previous data-driven DSMD methods largely overlooked the system interpretability, which is critical for a medical application, and they also suffered from the data sparsity issue at the same time. In this paper, we explore how to bring interpretability to data-driven DSMD. Specifically, we propose a more interpretable decision process to implement the dialogue manager of DSMD by reasonably mimicking real doctors' inquiry logics, and we devise a model with highly transparent components to conduct the inference. Moreover, we collect a new DSMD dataset, which has a much larger scale, more diverse patterns and is of higher quality than the existing ones. The experiments show that our method obtains 7.7%, 10.0%, 3.0% absolute improvement in diagnosis accuracy respectively on three datasets, demonstrating the effectiveness of its rational decision process and model design. Our codes and the GMD-12 dataset are available at https://github.com/lwgkzl/BR-Agent., Accepted by IJCAI 2022

While significant progress has been made in garment transfer, one of the most applicable directions of human-centric image generation, existing works overlook the in-the-wild imagery, presenting severe garment-person misalignment as well as noticeable degradation in fine texture details. This paper, therefore, attends to virtual try-on in real-world scenes and brings essential improvements in authenticity and naturalness especially for loose garment (e.g., skirts, formal dresses), challenging poses (e.g., cross arms, bent legs), and cluttered backgrounds. Specifically, we find that the pixel flow excels at handling loose garments whereas the vertex flow is preferred for hard poses, and by combining their advantages we propose a novel generative network called wFlow that can effectively push up garment transfer to in-the-wild context. Moreover, former approaches require paired images for training. Instead, we cut down the laboriousness by working on a newly constructed large-scale video dataset named Dance50k with self-supervised cross-frame training and an online cycle optimization. The proposed Dance50k can boost real-world virtual dressing by covering a wide variety of garments under dancing poses. Extensive experiments demonstrate the superiority of our wFlow in generating realistic garment transfer results for in-the-wild images without resorting to expensive paired datasets., Comment: Accepted at CVPR2022, Project: https://awesome-wflow.github.io

Recent advances in vision Transformers (ViTs) have come with a voracious appetite for computing power, high-lighting the urgent need to develop efficient training methods for ViTs. Progressive learning, a training scheme where the model capacity grows progressively during training, has started showing its ability in efficient training. In this paper, we take a practical step towards efficient training of ViTs by customizing and automating progressive learning. First, we develop a strong manual baseline for progressive learning of ViTs, by introducing momentum growth (MoGrow) to bridge the gap brought by model growth. Then, we propose automated progressive learning (AutoProg), an efficient training scheme that aims to achieve lossless acceleration by automatically increasing the training overload on-the-fly; this is achieved by adaptively deciding whether, where and how much should the model grow during progressive learning. Specifically, we first relax the optimization of the growth schedule to sub-network architecture optimization problem, then propose one-shot estimation of the sub-network performance via an elastic supernet. The searching overhead is reduced to minimal by recycling the parameters of the supernet. Extensive experiments of efficient training on ImageNet with two representative ViT models, DeiT and VOLO, demonstrate that AutoProg can accelerate ViTs training by up to 85.1% with no performance drop. Code: https://github.com/changlin31/AutoProg, Accepted to CVPR 2022

Recently, a surge of interest in visual transformers is to reduce the computational cost by limiting the calculation of self-attention to a local window. Most current work uses a fixed single-scale window for modeling by default, ignoring the impact of window size on model performance. How-ever, this may limit the modeling potential of these window-based models for multi-scale information. In this paper, we propose a novel method, named Dynamic Window Vision Transformer (DW-ViT). The dynamic window strategy proposed by DW- ViT goes beyond the model that employs a fixed single window setting. To the best of our knowl-edge, we are the first to use dynamic multi-scale windows to explore the upper limit of the effect of window settings on model performance. In DW- ViT, multi-scale information is obtained by assigning windows of different sizes to different head groups of window multi-head self-attention. Then, the information is dynamically fused by assigning different weights to the multi-scale window branches. We con-ducted a detailed performance evaluation on three datasets, ImageNet-1K, ADE20K, and COCO. Compared with re-lated state-of-the-art (SoTA) methods, DW- ViT obtains the best performance. Specifically, compared with the current SoTA Swin Transformers [31], DW-ViT has achieved con-sistent and substantial improvements on all three datasets with similar parameters and computational costs. In addition, DW-ViT exhibits good scalability and can be easily inserted into any window-based visual transformers.11Code release: https://github.com/pzhren/DW-ViT. This work was done when the first author interned at Dark Matter AI.

We present Laneformer, a conceptually simple yet powerful transformer-based architecture tailored for lane detection that is a long-standing research topic for visual perception in autonomous driving. The dominant paradigms rely on purely CNN-based architectures which often fail in incorporating relations of long-range lane points and global contexts induced by surrounding objects (e.g., pedestrians, vehicles). Inspired by recent advances of the transformer encoder-decoder architecture in various vision tasks, we move forwards to design a new end-to-end Laneformer architecture that revolutionizes the conventional transformers into better capturing the shape and semantic characteristics of lanes, with minimal overhead in latency. First, coupling with deformable pixel-wise self-attention in the encoder, Laneformer presents two new row and column self-attention operations to efficiently mine point context along with the lane shapes. Second, motivated by the appearing objects would affect the decision of predicting lane segments, Laneformer further includes the detected object instances as extra inputs of multi-head attention blocks in the encoder and decoder to facilitate the lane point detection by sensing semantic contexts. Specifically, the bounding box locations of objects are added into Key module to provide interaction with each pixel and query while the ROI-aligned features are inserted into Value module. Extensive experiments demonstrate our Laneformer achieves state-of-the-art performances on CULane benchmark, in terms of 77.1% F1 score. We hope our simple and effective Laneformer will serve as a strong baseline for future research in self-attention models for lane detection., AAAI2022

The perineurium serves as a selective, metabolically active diffusion barrier in the peripheral nervous system, which is composed of perineurial cells joined together by tight junctions (TJs). Not only are these junctions known to play an essential role in maintaining cellular polarity and tissue integrity, but also limit the paracellular diffusion of certain molecules and ions, whereas loss of TJs barrier function is imperative for tumour growth, invasion and metastasis. Hence, a detailed study on the barrier function of perineurial cells may provide insights into the molecular mechanism of perineural invasion (PNI). In this study, we aimed to develop an efficient procedure for the establishment of perineurial cell lines as a tool for investigating the physiology and pathophysiology of the peripheral nerve barriers. Herein, the isolation, expansion, characterization and maintenance of perineurial cell lines under favourable conditions are presented. Furthermore, the analysis of the phenotypic features of these perineurial cells as well as the barrier function for the study of PNI are described. Such techniques may provide a valuable means for the functional and molecular investigation of perineurial cells, and in particular may elucidate the pathogenesis and progression of PNI, and other peripheral nerve disorders.

Target-guided open-domain conversation aims to proactively and naturally guide a dialogue agent or human to achieve specific goals, topics or keywords during open-ended conversations. Existing methods mainly rely on single-turn datadriven learning and simple target-guided strategy without considering semantic or factual knowledge relations among candidate topics/keywords. This results in poor transition smoothness and low success rate. In this work, we adopt a structured approach that controls the intended content of system responses by introducing coarse-grained keywords, attains smooth conversation transition through turn-level supervised learning and knowledge relations between candidate keywords, and drives an conversation towards an specified target with discourse-level guiding strategy. Specially, we propose a novel dynamic knowledge routing network (DKRN) which considers semantic knowledge relations among candidate keywords for accurate next topic prediction of next discourse. With the help of more accurate keyword prediction, our keyword-augmented response retrieval module can achieve better retrieval performance and more meaningful conversations. Besides, we also propose a novel dual discourse-level target-guided strategy to guide conversations to reach their goals smoothly with higher success rate. Furthermore, to push the research boundary of target-guided open-domain conversation to match real-world scenarios better, we introduce a new large-scale Chinese target-guided open-domain conversation dataset (more than 900K conversations) crawled from Sina Weibo. Quantitative and human evaluations show our method can produce meaningful and effective target-guided conversations, significantly improving over other state-of-the-art methods by more than 20% in success rate and more than 0.6 in average smoothness score., 8 pages, 2 figues, 6tables, AAAI2020, fix our model's abbreviation

In this paper, we focus on triplet-based deep binary embedding networks for image retrieval task. The triplet loss has been shown to be effective for hashing retrieval. However, most of the triplet-based deep networks treat the triplets equally or select the hard triplets based on the loss. Such strategies do not consider the order relations of the binary codes and ignore the hash encoding when learning the feature representations. To this end, we propose an order-aware reweighting method to effectively train the triplet-based deep networks, which up-weights the important triplets and down-weights the uninformative triplets via the rank lists of the binary codes. First, we present the order-aware weighting factors to indicate the importance of the triplets, which depend on the rank order of binary codes. Then, we reshape the triplet loss to the squared triplet loss such that the loss function will put more weights on the important triplets. The extensive evaluations on several benchmark datasets show that the proposed method achieves significant performance compared with the state-of-the-art baselines.

Automatic generation of ophthalmic reports using data-driven neural networks has great potential in clinical practice. When writing a report, ophthalmologists make inferences with prior clinical knowledge. This knowledge has been neglected in prior medical report generation methods. To endow models with the capability of incorporating expert knowledge, we propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG), in which clinical relation triples are injected into the visual features as prior knowledge to drive the decoding procedure. However, two major common Knowledge Noise (KN) issues may affect models' effectiveness. 1) Existing general biomedical knowledge bases such as the UMLS may not align meaningfully to the specific context and language of the report, limiting their utility for knowledge injection. 2) Incorporating too much knowledge may divert the visual features from their correct meaning. To overcome these limitations, we design an automatic information extraction scheme based on natural language processing to obtain clinical entities and relations directly from in-domain training reports. Given a set of ophthalmic images, our CGT first restores a sub-graph from the clinical graph and injects the restored triples into visual features. Then visible matrix is employed during the encoding procedure to limit the impact of knowledge. Finally, reports are predicted by the encoded cross-modal features via a Transformer decoder. Extensive experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods and achieve state-of-the-art performances., CVPR 2022 (Poster)

Recently, a surge of interest in visual transformers is to reduce the computational cost by limiting the calculation of self-attention to a local window. Most current work uses a fixed single-scale window for modeling by default, ignoring the impact of window size on model performance. However, this may limit the modeling potential of these window-based models for multi-scale information. In this paper, we propose a novel method, named Dynamic Window Vision Transformer (DW-ViT). The dynamic window strategy proposed by DW-ViT goes beyond the model that employs a fixed single window setting. To the best of our knowledge, we are the first to use dynamic multi-scale windows to explore the upper limit of the effect of window settings on model performance. In DW-ViT, multi-scale information is obtained by assigning windows of different sizes to different head groups of window multi-head self-attention. Then, the information is dynamically fused by assigning different weights to the multi-scale window branches. We conducted a detailed performance evaluation on three datasets, ImageNet-1K, ADE20K, and COCO. Compared with related state-of-the-art (SoTA) methods, DW-ViT obtains the best performance. Specifically, compared with the current SoTA Swin Transformers \cite{liu2021swin}, DW-ViT has achieved consistent and substantial improvements on all three datasets with similar parameters and computational costs. In addition, DW-ViT exhibits good scalability and can be easily inserted into any window-based visual transformers.

Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context. Typically, existing approaches first retrieve external evidence and then perform commonsense reasoning using these evidence. In this paper, we propose a Self-supervised Bidirectional Encoder Representation Learning of Commonsense (elBERto) framework, which is compatible with off-the-shelf QA model architectures. The framework comprises five self-supervised tasks to force the model to fully exploit the additional training signals from contexts containing rich commonsense. The tasks include a novel Contrastive Relation Learning task to encourage the model to distinguish between logically contrastive contexts, a new Jigsaw Puzzle task that requires the model to infer logical chains in long contexts, and three classic SSL tasks to maintain pre-trained models language encoding ability. On the representative WIQA, CosmosQA, and ReClor datasets, elBERto outperforms all other methods, including those utilizing explicit graph reasoning and external knowledge retrieval. Moreover, elBERto achieves substantial improvements on out-of-paragraph and no-effect questions where simple lexical similarity comparison does not help, indicating that it successfully learns commonsense and is able to leverage it when given dynamic context.

The whale optimization algorithm (WOA) is a popular swarm intelligence algorithm which simulates the hunting behavior of humpback whales. WOA has the deficiency of easily falling into the local optimal solutions. In order to overcome the weakness of the WOA, a modified variant of WOA called OCDWOA is proposed. There are four main operators introduced into the OCDWOA to enhance the search performance of WOA. The operators include opposition-based learning method, nonlinear parameter design, density peak clustering strategy, and differential evolution. The proposed algorithm is tested on 19 optimization benchmark functions and a seismic inversion problem. OCDWOA is compared with the classical WOA and three typical variants of WOA. The results demonstrate that OCDWOA outperforms the compared algorithms in terms of obtaining the global optimal solution.

Neural Architecture Search (NAS) aims to find efficient models for multiple tasks. Beyond seeking solutions for a single task, there are surging interests in transferring network design knowledge across multiple tasks. In this line of research, effectively modeling task correlations is vital yet highly neglected. Therefore, we propose \textbf{Arch-Graph}, a transferable NAS method that predicts task-specific optimal architectures with respect to given task embeddings. It leverages correlations across multiple tasks by using their embeddings as a part of the predictor's input for fast adaptation. We also formulate NAS as an architecture relation graph prediction problem, with the relational graph constructed by treating candidate architectures as nodes and their pairwise relations as edges. To enforce some basic properties such as acyclicity in the relational graph, we add additional constraints to the optimization process, converting NAS into the problem of finding a Maximal Weighted Acyclic Subgraph (MWAS). Our algorithm then strives to eliminate cycles and only establish edges in the graph if the rank results can be trusted. Through MWAS, Arch-Graph can effectively rank candidate models for each task with only a small budget to finetune the predictor. With extensive experiments on TransNAS-Bench-101, we show Arch-Graph's transferability and high sample efficiency across numerous tasks, beating many NAS methods designed for both single-task and multi-task search. It is able to find top 0.16\% and 0.29\% architectures on average on two search spaces under the budget of only 50 models., Comment: Accepted by CVPR 2022

Aiming at the shortcomings of the Harris hawks optimization algorithm (HHO), such as poor initial population diversity, slow convergence speed, poor local optimization ability, and easily falling into local optimum, a Harris hawks optimization algorithm (CCCHHO) integrating multiple mechanisms is proposed. First, the population diversity is enhanced by the initialization of the chaotic method. Second, the cosine function is used to better simulate the characteristics of the periodic change of the energy of the prey in the repeated contests with the group of hawks, to better balance the exploration and exploitation of the algorithm. Third, Cauchy mutation on the optimal individual in the exploration phase is performed, and the characteristics of the Cauchy distribution to enhance the diversity of the population are used, which can effectively prevent the algorithm from falling into the local optimum. Fourth, the local optimization ability of the algorithm by using the ergodicity of the chaotic system in the exploitation phase to perform a chaotic local search for the optimal individual is enhanced, which can effectively jump out after the algorithm falls into the local optimum. Finally, we use the elite individuals of the population to guide the position update of the population’s individuals, fully communicate with the dominant individuals, and speed up the convergence speed of the algorithm. Through the simulation experiments on CCCHHO with 11 different benchmark functions, CCCHHO is better than the gray wolf optimization algorithm (GWO), the Salp swarm algorithm (SSA), the ant lion optimization algorithm (ALO), and three improved HHO algorithms in terms of convergence speed and optimization accuracy, whether it is a unimodal benchmark function or a multimodal benchmark function. The experimental results show that CCCHHO has excellent algorithm efficiency and robustness.

Despite the significant success in various domains, the data-driven deep neural networks compromise the feature interpretability, lack the global reasoning capability, and can’t incorporate external information crucial for complicated real-world tasks. Since the structured knowledge can provide rich cues to record human observations and commonsense, it is thus desirable to bridge symbolic semantics with learned local feature representations. In this chapter, we review works that incorporate different domain knowledge into the intermediate feature representation.These methods firstly construct a domain-specific graph that represents related human knowledge. Then, they characterize node representations with neural network features and perform graph convolution to enhance these symbolic nodes via the graph neural network(GNN).Lastly, they map the enhanced node feature back into the neural network for further propagation or prediction. Through integrating knowledge graphs into neural networks, one can collaborate feature learning and graph reasoning with the same supervised loss function and achieve a more effective and interpretable way to introduce structure constraints.

Transferring human motion from a source to a target person poses great potential in computer vision and graphics applications. A crucial step is to manipulate sequential future motion while retaining the appearance characteristic.Previous work has either relied on crafted 3D human models or trained a separate model specifically for each target person, which is not scalable in practice.This work studies a more general setting, in which we aim to learn a single model to parsimoniously transfer motion from a source video to any target person given only one image of the person, named as Collaborative Parsing-Flow Network (CPF-Net). The paucity of information regarding the target person makes the task particularly challenging to faithfully preserve the appearance in varying designated poses. To address this issue, CPF-Net integrates the structured human parsing and appearance flow to guide the realistic foreground synthesis which is merged into the background by a spatio-temporal fusion module. In particular, CPF-Net decouples the problem into stages of human parsing sequence generation, foreground sequence generation and final video generation. The human parsing generation stage captures both the pose and the body structure of the target. The appearance flow is beneficial to keep details in synthesized frames. The integration of human parsing and appearance flow effectively guides the generation of video frames with realistic appearance. Finally, the dedicated designed fusion network ensure the temporal coherence. We further collect a large set of human dancing videos to push forward this research field. Both quantitative and qualitative results show our method substantially improves over previous approaches and is able to generate appealing and photo-realistic target videos given any input person image. All source code and dataset will be released at https://github.com/xiezhy6/CPF-Net., Comment: TIP 2021

Two kinds of porous structure design strategies, ring-support (RS) and column-support (CS), are proposed for human implants. The accurate design of porosity is realized by adjusting the pore charac...

Despite recent progress on image-based virtual try-on, current methods are constraint by shared warping networks and thus fail to synthesize natural try-on results when faced with clothing categories that require different warping operations. In this paper, we address this problem by finding clothing category-specific warping networks for the virtual try-on task via Neural Architecture Search (NAS). We introduce a NAS-Warping Module and elaborately design a bilevel hierarchical search space to identify the optimal network-level and operation-level flow estimation architecture. Given the network-level search space, containing different numbers of warping blocks, and the operation-level search space with different convolution operations, we jointly learn a combination of repeatable warping cells and convolution operations specifically for the clothing-person alignment. Moreover, a NAS-Fusion Module is proposed to synthesize more natural final try-on results, which is realized by leveraging particular skip connections to produce better-fused features that are required for seamlessly fusing the warped clothing and the unchanged person part. We adopt an efficient and stable one-shot searching strategy to search the above two modules. Extensive experiments demonstrate that our WAS-VTON significantly outperforms the previous fixed-architecture try-on methods with more natural warping results and virtual try-on results.

We present Voxel Transformer (VoTr), a novel and effective voxel-based Transformer backbone for 3D object detection from point clouds. Conventional 3D convolutional backbones in voxel-based 3D detectors cannot efficiently capture large context information, which is crucial for object recognition and localization, owing to the limited receptive fields. In this paper, we resolve the problem by introducing a Transformer-based architecture that enables long-range relationships between voxels by self-attention. Given the fact that non-empty voxels are naturally sparse but numerous, directly applying standard Transformer on voxels is non-trivial. To this end, we propose the sparse voxel module and the submanifold voxel module, which can operate on the empty and non-empty voxel positions effectively. To further enlarge the attention range while maintaining comparable computational overhead to the convolutional counterparts, we propose two attention mechanisms for multi-head attention in those two modules: Local Attention and Dilated Attention, and we further propose Fast Voxel Query to accelerate the querying process in multi-head attention. VoTr contains a series of sparse and submanifold voxel modules and can be applied in most voxel-based detectors. Our proposed VoTr shows consistent improvement over the convolutional baselines while maintaining computational efficiency on the KITTI dataset and the Waymo Open dataset., To appear at ICCV 2021