Search

Previous studies have constantly observed that a language model repeats itself, creating repetitions in an output sequence. To cope with the issue, stochastic decoding schemes have been the de facto approaches; the strategies add randomness in inference, hence avoiding the “self-loop”. However, the remedy comes at the cost of sacrificing output quality due to the randomness involved. In this work, we introduce a deterministic decoding scheme, local temperature beam search. This inference algorithm is an embarrassingly simple variant of beam search, yet it reduces repetition, whose level is superior to that of a sampling-based decoding algorithm, while maintaining the level of coherence as in beam search. Our idea is rooted in the concept of model calibration; we view a repetition as a casualty from overconfidence in a model. Therefore, our work mitigates the miscalibration present in the course of inference with a post-calibration approach applied in beam-specific manner. Our inference scheme is validated on text completion tasks, in which the repetition problem is seen most clearly, and is exhaustively compared with existing inference schemes. © 2023 Association for Computational Linguistics.

Despite the popularity of Vision Transformers (ViTs) and eXplainable AI (XAI), only a few explanation methods have been designed specially for ViTs thus far. They mostly use attention weights of the [CLS] token on patch embeddings and often produce unsatisfactory saliency maps. This paper proposes a novel method for explaining ViTs called ViT-CX. It is based on patch embeddings, rather than attentions paid to them, and their causal impacts on the model output. Other characteristics of ViTs such as causal overdetermination are also considered in the design of ViT-CX. The empirical results show that ViT-CX produces more meaningful saliency maps and does a better job revealing all important evidence for the predictions than previous methods. The explanation generated by ViT-CX also shows significantly better faithfulness to the model. The codes and appendix are available at https://github.com/vaynexie/CausalX-ViT. © 2023 International Joint Conferences on Artificial Intelligence. All rights reserved.

The need to explain the output of a deep neural network classifier is now widely recognized. While previous methods typically explain a single class in the output, we advocate explaining the whole output, which is a probability distribution over multiple classes. A whole-output explanation can help a human user gain an overall understanding of model behaviour instead of only one aspect of it. It can also provide a natural framework where one can examine the evidence used to discriminate between competing classes, and thereby obtain contrastive explanations. In this paper, we propose a contrastive whole-output explanation (CWOX) method for image classification, and evaluate it using quantitative metrics and through human subject studies. The source code of CWOX is available at https://github.com/vaynexie/CWOX. © UAI 2023. All rights reserved.

The goal of document-grounded dialogue (DocGD) is to generate a response by anchoring the evidence in a supporting document in accordance with the dialogue context. This entails four causally interconnected variables. While task-specific pre-training has significantly enhanced performances on numerous downstream tasks, existing DocGD methods still rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To address this, we present the first causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora. Additionally, we propose a causally-perturbed pre-training strategy to better capture causality by introducing perturbations on the variables and optimizing the overall causal effect. Experiments conducted on three benchmark datasets demonstrate that our causal pre-training yields substantial and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings.

Conversational models aim to generate readable textual responses to input queries. In recent years, conversational models are typically built using deep neural networks, which are called deep conversational models (DCMs). DCMs can be used for task-oriented dialogues or chit-chat conversations. In this thesis, we investigate ways to enhance DCMs for chit-chat conversations via input augmentation and data source expansion. DCM maps an input query to some output responses. It has been observed in previous work that the output is often uninformative and lacks diversity due to two reasons: (1) The input does not contain sufficient information to determine an appropriate output, and (2) the DCM is trained via likelihood maximization and hence captures only the most salient input-output relationships. One common method to address the first issue is to include background documents as additional inputs to the DCM, and one popular way to deal with the second issue is to include retrieved responses to similar input queries as additional inputs to the DCM. In this thesis, we advance the state-of-the-art in both of those two lines of work. For the first line of work, we propose an output-anticipated memory module to enable the DCM to better attend to the relevant information in the background documents. For the second line of work, we develop a memory module to extract relationships between clusters of similar inputs and clusters of outputs (which are more robust than relationships between individual inputs and outputs), and use the relationships to improve the performance of the DCM. Nowadays, DCMs are often trained in huge corpora. However, there are still scenarios with low resources. One example is an online chatbot that needs to quickly adapt to a new user after a few rounds of conversations. Our third contribution in this thesis is a meta-learning-based method to help with the adaptation by utilizing data from the user's friends, who are expected to have similar interests

Overconfidence has been shown to impair generalization and calibration of a neural network. Previous studies remedy this issue by adding a regularization term to a loss function, preventing a model from making a peaked distribution. Label smoothing smoothes target labels with a pre-defined prior label distribution; as a result, a model is learned to maximize the likelihood of predicting the soft label. Nonetheless, the amount of smoothing is the same in all samples and remains fixed in training. In other words, label smoothing does not reflect the change in probability distribution mapped by a model over the course of training. To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by taking model probability distribution into account, thereby varying the parameter per instance. A model in training self-regulates the extent of smoothing on the fly during forward propagation. Furthermore, inspired by recent work in bridging label smoothing and knowledge distillation, our work utilizes self-knowledge as a prior label distribution in softening target labels, and presents theoretical support for the regularization effect by knowledge distillation and the dynamic smoothing parameter. Our regularizer is validated comprehensively, and the result illustrates marked improvements in model generalization and calibration, enhancing robustness and trustworthiness of a model.

In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that holds inter-class relations which send a meaningful supervision to a student; hence, much effort has been put to find such knowledge to be distilled. In this paper, we explore a question that has been given little attention: “when to distill such knowledge.” The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscalibration of a student. This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data. We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level. Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model generalization, but also significantly lowers model calibration error.

Protecting the privacy of user data is crucial for text generation models, which can leak sensitive information during generation. Differentially private (DP) learning methods provide guarantees against identifying the existence of a training sample from model outputs. PATE is a recent DP learning algorithm that achieves high utility with strong privacy protection on training samples. However, text generation models output tokens sequentially in a large output space; the classic PATE algorithm is not customized for this setting. Furthermore, PATE works well to protect sample-level privacy, but is not designed to protect phrases in samples. In this paper, we propose SeqPATE, an extension of PATE to text generation that protects the privacy of individual training samples and sensitive phrases in training data. To adapt PATE to text generation, we generate pseudo-contexts and reduce the sequence generation problem to a next-word prediction problem. To handle the large output space, we propose a candidate filtering strategy to dynamically reduce the output space, and refine the teacher aggregation of PATE to avoid low agreement due to voting for a large number of candidates. To further reduce privacy losses, we use knowledge distillation to reduce the number of teacher queries. The experiments verify the effectiveness of SeqPATE in protecting both training samples and sensitive phrases.

Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored. In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data. We propose an unlabeled data enhanced lifelong learner to explore SSLL. Specially, we dedicate task-specific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks. Experimental results on various language tasks demonstrate our model's effectiveness and superiority over competitive baselines under the new setting SSLL. We will release the code and data.

Emotional conversation systems generate responses for the input queries considering the speaker’s emotions in a conversation. Existing emotional conversation systems output emotional responses according to either a given emotion or the user’s emotion reflected in the input queries. Following a given emotion may lead to an emotional drift between the given emotion and the conversation state, and following only the user’s emotion may aggravate the user’s negative feelings if users suffer from a negative mood. In this paper, we propose to generate empathetic responses catering to the user’s emotions while leading the conversation to be emotionally positive. Particularly, by abstracting the conversation corpus, we extract and store the different responding strategies for different users’ emotions and conversational topics into a memory. We encourage positive emotions in conversation via a sentiment evaluator. We model the memory outputs with a Gaussian mixture distribution and sample a final responding strategy from the distribution. The strategy acts as a condition to a transformer model to generate responses. The experiments verify our model surpasses the baseline methods in appropriateness, diversity, and generating emotionally positive responses.

Lifelong learning (LL) is vital for advanced task-oriented dialogue (ToD) systems. To address the catastrophic forgetting issue of LL, generative replay methods are widely employed to consolidate past knowledge with generated pseudo samples. However, most existing generative replay methods use only a single task-specific token to control their models. This scheme is usually not strong enough to constrain the generative model due to insufficient information involved. In this paper, we propose a novel method, prompt conditioned VAE for lifelong learning (PCLL), to enhance generative replay by incorporating tasks’ statistics. PCLL captures task-specific distributions with a conditional variational autoencoder, conditioned on natural language prompts to guide the pseudo-sample generation. Moreover, it leverages a distillation process to further consolidate past knowledge by alleviating the noise in pseudo samples. Experiments on natural language understanding tasks of ToD systems demonstrate that PCLL significantly outperforms competitive baselines in building lifelong learning models.

Building models of natural language processing (NLP) is challenging in low-resource scenarios where only limited data are available. Optimization-based meta-learning algorithms achieve promising results in low-resource scenarios by adapting a well-generalized model initialization to handle new tasks. Nonetheless, these approaches suffer from the memorization overfitting issue, where the model tends to memorize the meta-training tasks while ignoring support sets when adapting to new tasks. To address this issue, we propose a memory imitation meta-learning (MemIML) method that enhances the model’s reliance on support sets for task adaptation. Specifically, we introduce a task-specific memory module to store support set information and construct an imitation module to force query sets to imitate the behaviors of some representative supportset samples stored in the memory. A theoretical analysis is provided to prove the effectiveness of our method, and empirical results also demonstrate that our method outperforms competitive baselines on both text classification and generation tasks.

Background: A supplementary chapter on the diagnostic patterns of Traditional Medicine, including Traditional Chinese Medicine (TCM), was introduced into the latest edition of the International Classification of Diseases (ICD-11). However, evidence-based rules are yet to be developed for pattern differentiation in patients with specific conventional medicine diagnoses. Without such standardised rules, the level of diagnostic agreement amongst practitioners is unsatisfactory. This may reduce the reliability of practice and the generalisability of clinical research. Purpose: Using cross-sectional study data from patients with functional dyspepsia, we reviewed and illustrated a quantitative approach that combines TCM expertise and computer algorithmic capacity, namely latent tree analysis (LTA), to establish score-based pattern differentiation rules. Review of methods: LTA consists of six major steps: (i) the development of a TCM clinical feature questionnaire; (ii) statistical pattern discovery; (iii) statistical pattern interpretation; (iv) TCM diagnostic pattern identification; (v) TCM diagnostic pattern quantification; and (vi) TCM diagnostic pattern differentiation. Step (i) involves the development of a comprehensive questionnaire covering all essential TCM clinical features of the disease of interest via a systematic review. Step (ii) to (iv) required input from TCM experts, with the algorithmic capacity provided by Lantern, a dedicated software for TCM LTA. Motivational example to illustrate the methods: LTA is used to quantify the diagnostic importance of various clinical features in each TCM diagnostic pattern in terms of mutual information and cumulative information coverage. LTA is also capable of deriving score-based differentiation rules for each TCM diagnostic pattern, with each clinical feature being provided with a numerical score for its presence. Subsequently, a summative threshold is generated to allow pattern differentiation. If the total score of

Background Traditional Chinese Medicine (TCM) treatment strategies are guided by pattern differentiation, as documented in the eleventh edition of the International Classification of Diseases (ICD). However, no standards for pattern differentiation are proposed to ensure inter-rater agreement. Without standardisation, research on associations between TCM diagnostic patterns, clinical features, and geographical characteristics is also not feasible. This diagnostic cross-sectional study aimed to (i) establish the pattern differentiation rules of functional dyspepsia (FD) using latent tree analysis (LTA); (ii) compare the prevalence of diagnostic patterns in Hong Kong and Hunan; (iii) discover the co-existence of diagnostic patterns; and (iv) reveal the associations between diagnostic patterns and FD common comorbidities. Methods A total of 250 and 150 participants with FD consecutively sampled in Hong Kong and Hunan, respectively, completed a questionnaire on TCM clinical features. LTA was performed to reveal TCM diagnostic patterns of FD and derive relevant pattern differentiation rules. Multivariate regression analyses were performed to quantify correlations between different diagnostic patterns and between diagnostic patterns and clinical and geographical variables. Results At least one TCM diagnostic pattern was differentiated in 70.7%, 73.6%, and 64.0% of the participants in the overall (n = 400), Hong Kong (n = 250), and Hunan (n = 150) samples, respectively, using the eight pattern differentiation rules derived. 52.7% to 59.6% of the participants were diagnosed with two or more diagnostic patterns. Cold-heat complex (59.8%) and spleen-stomach dampness-heat (77.1%) were the most prevalent diagnostic patterns in Hong Kong and Hunan, respectively. Spleen-stomach deficiency cold was highly likely to co-exist with spleen-stomach qi deficiency (adjusted odds ratio (AOR): 53.23; 95% confidence interval (CI): 21.77 to 130.16). Participants with severe anxiety tended to

Objective: To treat patients with psoriasis vulgaris using Traditional Chinese Medicine (TCM), one must stratify patients into subtypes (known as TCM syndromes or Zheng) and apply appropriate TCM treatments to different subtypes. However, no unified symptom-based classification scheme of subtypes (Zheng) exists for psoriasis vulgaris. The present paper aims to classify patients with psoriasis vulgaris into different subtypes via the analysis of clinical TCM symptom and sign data. Methods: A cross-sectional survey was carried out in Beijing from 2005-2008, collecting clinical TCM symptom and sign data from 2,764 patients with psoriasis vulgaris. Roughly 108 symptoms and signs were initially analyzed using latent tree analysis, with a selection of the resulting latent variables then used as features to cluster patients into subtypes. Results: The initial latent tree analysis yielded a model with 43 latent variables. The second phase of the analysis divided patients into three subtype groups with clear TCM Zheng connotations: ‘blood deficiency and wind dryness’; ‘blood heat’; and ‘blood stasis’. Conclusions: Via two-phase analysis of clinic symptom and sign data, three different Zheng subtypes were identified for psoriasis vulgaris. Statistical characteristics of the three subtypes are presented. This constitutes an evidence-based solution to the syndrome-differentiation problem that exists with psoriasis vulgaris.

Attention is an essential mechanism for creatures to perceive the world. Psychologists describe it as the allocation of limited cognitive processing resources. Neural attention is a technique motivated by cognitive attention. It selectively processes information from sources by computing input-dependent dynamic weights to boost the information from relevant portions. Neural attention was first proposed in deep learning in 2014 and underwent extensive developments in the decade. A milestone in the development is the proposal of Transformers in 2017. The transformer is the first deep architecture constructed solely on attention mechanisms without any recurrence and convolution. Attention as led to great success in many areas, such as natural language processing, computer vision, and social networks, and has become an essential component in neural networks. While much research on attention focuses exploring its uses in ever-increasing applications, there is also significant interest in enhancing attention in current models. The motivations are to avoid networks being distracted by irrelevant information and to improve the network’s interpretability. Existing works fall into two categories. One enhances attention by purely improving attention mechanisms and the other enhances attention by exploring patterns in data. In this thesis, we focus on enhancing attention in deep natural language processing (NLP) models. The contributions of this thesis are as follows: First, we propose Gated Attention Network (GA-Net), a novel sparse attention network, for sequence data. GA-Net combines the techniques of attention and dynamic network configuration. It dynamically selects a subset of elements to pay attention to and filters irrelevant elements. Besides, an efficient end-to-end learning method using Gumbel-softmax is designed to relax the binary gates and enable back-propagation to facilitate GA-Net training. GA-Net achieves better performance in text classification tasks compare

objective: To treat patients with psoriasis vulgaris using Traditional Chinese Medicine (TCM), one must stratify patients into subtypes (known as TCM syndromes or Zheng) and apply appropriate TCM treatments to different subtypes. However, no unified symptom-based classification scheme of subtypes (Zheng) exists for psoriasis vulgaris. The present paper aims to classify patients with psoriasis vulgaris into different subtypes via the analysis of clinical TCM symptom and sign data. Methods: A cross-sectional survey was carried out in Beijing from 2005-2008, collecting clinical TCM symptom and sign data from 2,764 patients with psoriasis vulgaris. Roughly 108 symptoms and signs were initially analyzed using latent tree analysis, with a selection of the resulting latent variables then used as features to cluster patients into subtypes. Results: The initial latent tree analysis yielded a model with 43 latent variables. The second phase of the analysis divided patients into three subtype groups with clear TCM Zheng connotations: ‘blood deficiency and wind dryness’; ‘blood heat’; and ‘blood stasis’. Conclusions: Via two-phase analysis of clinic symptom and sign data, three different Zheng subtypes were identified for psoriasis vulgaris. Statistical characteristics of the three subtypes are presented. This constitutes an evidence-based solution to the syndrome-differentiation problem that exists with psoriasis vulgaris.

Rap generation, which aims to produce lyrics and corresponding singing beats, needs to model both rhymes and rhythms. Previous works for rap generation focused on rhyming lyrics, but ignored rhythmic beats, which are important for rap performance. In this paper, we develop DeepRapper, a Transformer-based rap generation system that can model both rhymes and rhythms. Since there is no available rap datasets with rhythmic beats, we develop a data mining pipeline to collect a large-scale rap dataset, which includes a large number of rap songs with aligned lyrics and rhythmic beats. Second, we design a Transformer-based autoregressive language model which carefully models rhymes and rhythms. Specifically, we generate lyrics in the reverse order with rhyme representation and constraint for rhyme enhancement, and insert a beat symbol into lyrics for rhythm/beat modeling. To our knowledge, DeepRapper is the first system to generate rap with both rhymes and rhythms. Both objective and subjective evaluations demonstrate that DeepRapper generates creative and high-quality raps with rhymes and rhythms.

Text style transfer aims to alter the style (e.g., sentiment) of a sentence while preserving its content. A common approach is to map a given sentence to content representation that is free of style, and the content representation is fed to a decoder with a target style. Previous methods in filtering style completely remove tokens with style at the token level, which incurs the loss of content information. In this paper, we propose to enhance content preservation by implicitly removing the style information of each token with reverse attention, and thereby retain the content. Furthermore, we fuse content information when building the target style representation, making it dynamic with respect to the content. Our method creates not only style-independent content representation, but also content-dependent style representation in transferring style. Empirical results show that our method outperforms the state-of-the-art baselines by a large margin in terms of content preservation. In addition, it is also competitive in terms of style transfer accuracy and fluency.

Background: Previous UK Biobank studies showed that symptoms and physical measurements had excellent prediction on long-term clinical outcomes in general population. Symptoms and signs could intuitively and non-invasively predict and monitor disease progression, especially for telemedicine, but related research is limited in diabetes and renal medicine. Methods: This retrospective cohort study aimed to evaluate the predictive power of a symptom-based stratification framework and individual symptoms for diabetes. Three hundred two adult diabetic patients were consecutively sampled from outpatient clinics in Hong Kong for prospective symptom assessment. Demographics and longitudinal measures of biochemical parameters were retrospectively extracted from linked medical records. The association between estimated glomerular filtration rate (GFR) (independent variable) and biochemistry, epidemiological factors, and individual symptoms was assessed by mixed regression analyses. A symptom-based stratification framework of diabetes using symptom clusters was formulated by Delphi consensus method. Akaike information criterion (AIC) and Bayesian information criterion (BIC) were compared between statistical models with different combinations of biochemical, epidemiological, and symptom variables. Results: In the 4.2-year follow-up period, baseline presentation of edema (-1.8 ml/min/1.73m(2), 95%CI: -2.5 to -1.2, p < 0.001), epigastric bloating (-0.8 ml/min/1.73m(2), 95%CI: -1.4 to -0.2, p = 0.014) and alternating dry and loose stool (-1.1 ml/min/1.73m(2), 95%CI: -1.9 to -0.4, p = 0.004) were independently associated with faster annual GFR decline. Eleven symptom clusters were identified from literature, stratifying diabetes predominantly by gastrointestinal phenotypes. Using symptom clusters synchronized by Delphi consensus as the independent variable in statistical models reduced complexity and improved explanatory power when compared to using individual symptoms. Symptom-biolo

Personalized conversation models (PCMs) generate responses according to speaker preferences. Existing personalized conversation tasks typically require models to extract speaker preferences from user descriptions or their conversation histories, which are scarce for newcomers and inactive users. In this paper, we propose a few-shot personalized conversation task with an auxiliary social network. The task requires models to generate personalized responses for a speaker given a few conversations from the speaker and a social network. Existing methods are mainly designed to incorporate descriptions or conversation histories. Those methods can hardly model speakers with so few conversations or connections between speakers. To better cater for newcomers with few resources, we propose a personalized conversation model (PCM) that learns to adapt to new speakers as well as enabling new speakers to learn from resource-rich speakers. Particularly, based on a meta-learning based PCM, we propose a task aggregator (TA) to collect other speakers' information from the social network. The TA provides prior knowledge of the new speaker in its meta-learning. Experimental results show our methods outperform all baselines in appropriateness, diversity, and consistency with speakers.

Background: Pattern diagnosis-guided treatments in Traditional Chinese Medicine (TCM) has been recognised by the eleventh revision of the International Classification of Diseases (ICD-11). Accurate pattern diagnosis requires reliable and valid diagnostic instruments that guide the collection of TCM clinical data without bias. This study synthesised the existing TCM diagnostic instruments for functional dyspepsia (FD) and appraised their quality regarding their development process and measurement properties. Methods: Seven electronic databases were searched for validation studies on TCM diagnostic instruments for FD. Synthesis and appraisal of the included studies were performed following the COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) Initiative guidelines adapted for TCM diagnostic instruments. Risk of bias assessment was conducted using the COSMIN Risk of Bias Checklist. Results: Five studies were included, with five unique TCM diagnostic instruments for FD identified. All five diagnostic instruments were of inadequate quality in terms of their development process, implying a shortcoming in their relevance, comprehensibility, and comprehensiveness. Only the criterion validity of Stomach Qi Deficiency Pattern Assessment Scale was of sufficient quality and had no risk of bias in its validation. Conclusion: The quality of TCM diagnostic instruments for FD warrants urgent improvements. None of them was considered reliable or valid for guiding TCM pattern diagnosis. To support the evidence base of the standardization of TCM patterns in ICD-11, TCM diagnostic instruments should be developed and validated rigorously under the COSMIN guidelines. Amendments should be made on the guidelines to accommodate the features and uniqueness of TCM diagnostic process.

Discriminative clustering approaches assign data points to different groups by identifying sparse regions, without explicitly modeling the dataset and categories. Such methods are flexible and powerful in practice since they make few assumptions. In particular, the probabilistic-based Softmax model makes only one assumption, which is that data points are linearly separable. Therefore, it is potentially suitable in clustering data processed by feature transformation techniques. The principle of cluster assumption states that decision boundaries of clusters should lie in low-density regions. In previous works on discriminative clustering, this principle has been compromised by the cluster balance consideration, which is incorporated to avoid degenerate clustering solutions. However, datasets are rarely balanced with respect to attributes of interest. Furthermore, large clusters from imbalanced datasets might also contain sparse regions, where decision boundaries should not be positioned. In this thesis, we present self-optimality, a novel criterion for Softmax discriminative clustering, which is faithful to the cluster assumption principle and is free of cluster balance considerations. We also propose an adaptive algorithm aimed at finding self-optimal solutions, which can accurately recognize clusters from linearly separable imbalanced datasets with multiple degrees of sparseness.

Background: Pattern diagnosis-guided treatments in Traditional Chinese Medicine (TCM) has been recognised by the eleventh revision of the International Classification of Diseases (ICD-11). Accurate pattern diagnosis requires reliable and valid diagnostic instruments that guide the collection of TCM clinical data without bias. This study synthesised the existing TCM diagnostic instruments for functional dyspepsia (FD) and appraised their quality regarding their development process and measurement properties. Methods: Seven electronic databases were searched for validation studies on TCM diagnostic instruments for FD. Synthesis and appraisal of the included studies were performed following the COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) Initiative guidelines adapted for TCM diagnostic instruments. Risk of bias assessment was conducted using the COSMIN Risk of Bias Checklist. Results: Five studies were included, with five unique TCM diagnostic instruments for FD identified. All five diagnostic instruments were of inadequate quality in terms of their development process, implying a shortcoming in their relevance, comprehensibility, and comprehensiveness. Only the criterion validity of Stomach Qi Deficiency Pattern Assessment Scale was of sufficient quality and had no risk of bias in its validation. Conclusion: The quality of TCM diagnostic instruments for FD warrants urgent improvements. None of them was considered reliable or valid for guiding TCM pattern diagnosis. To support the evidence base of the standardization of TCM patterns in ICD-11, TCM diagnostic instruments should be developed and validated rigorously under the COSMIN guidelines. Amendments should be made on the guidelines to accommodate the features and uniqueness of TCM diagnostic process.

Neural conversation models are known to generate appropriate but non-informative responses in general. A scenario where informativeness can be significantly enhanced is Conversing by Reading (CbR), where conversations take place with respect to a given external document. In previous work, the external document is utilized by (1) creating a context-aware document memory that integrates information from the document and the conversational context, and then (2) generating responses referring to the memory. In this paper, we propose to create the document memory with some anticipated responses in mind. This is achieved using a teacher-student framework. The teacher is given the external document, the context, and the ground-truth response, and learns how to build a response-aware document memory from three sources of information. The student learns to construct a response-anticipated document memory from the first two sources, and the teacher's insight on memory creation. Empirical results show that our model outperforms the previous state-of-the-art for the CbR task.

Although deep neural networks generally have fixed network structures, the concept of dynamic mechanism has drawn more and more attention in recent years. Attention mechanisms compute input-dependent dynamic attention weights for aggregating a sequence of hidden states. Dynamic network configuration in convolutional neural networks (CNNs) selectively activates only part of the network at a time for different inputs. In this paper, we combine the two dynamic mechanisms for text classification tasks. Traditional attention mechanisms attend to the whole sequence of hidden states for an input sentence, while in most cases not all attention is needed especially for long sequences. We propose a novel method called Gated Attention Network (GA-Net) to dynamically select a subset of elements to attend to using an auxiliary network, and compute attention weights to aggregate the selected elements. It avoids a significant amount of unnecessary computation on unattended elements, and allows the model to pay attention to important parts of the sequence. Experiments in various datasets show that the proposed method achieves better performance compared with all baseline models with global or local attention while requiring less computation and achieving better interpretability. It is also promising to extend the idea to more complex attention-based models, such as transformers and seq-to-seq models.

Due to their designable properties, microstructural materials have emerged as an important class of materials that have the potential for used in a variety of applications. The design of such materials is challenged by the multifunctionality requirements and various constraints stemmed from manufacturing limitations and other practical considerations. Traditional design methods such as those based on topological optimization techniques rely heavily on high-dimensional physical simulations and can be inefficient. In addition, it is difficult to impose geometrical constraints in those methods. In this work, we propose a deep learning model based on deep convolutional generative adversarial network (DCGAN) and convolutional neural network (CNN) for the design of microstructural materials. The DCGAN is used to generate design candidates that satisfy geometrical constraints and the CNN is used as a surrogate model to link the microstructure to its properties. Once trained, the two networks are combined to form the design network which is utilized to for the inverse design. The advantages of the method include its high efficiency and the simplicity in handling geometrical constraints. In addition, no high-dimensional sensitivity simulations are required. The performance of the method is demonstrated on the design of microstructural materials with desired compliance tensor, subject to specified geometrical constraints. © 2019, Springer-Verlag GmbH Germany, part of Springer Nature.

Neural conversation models are known to generate appropriate but non-informative responses in general. A scenario where informativeness can be significantly enhanced is Conversing by Reading (CbR), where conversations take place with respect to a given external document. In previous work, the external document is utilized by (1) creating a context-aware document memory that integrates information from the document and the conversational context, and then (2) generating responses referring to the memory. In this paper, we propose to create the document memory with some anticipated responses in mind. This is achieved using a teacher-student framework. The teacher is given the external document, the context, and the ground-truth response, and learns how to build a response-aware document memory from three sources of information. The student learns to construct a response-anticipated document memory from the first two sources, and the teacher's insight on memory creation. Empirical results show that our model outperforms the previous state-of-the-art for the CbR task.

Neural generative models for open-domain chit-chat conversations have become an active area of research in recent years. A critical issue with most existing generative models is that the generated responses lack informativeness and diversity. A few researchers attempt to leverage the results of retrieval models to strengthen the generative models, but these models are limited by the quality of the retrieval results. In this work, we propose a memory-augmented generative model, which learns to abstract from the training corpus and saves the useful information to the memory to assist the response generation. Our model clusters query-response samples, extracts characteristics of each cluster, and learns to utilize these characteristics for response generation. Experimental results show that our model outperforms other competitive baselines.

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features. In general, our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data. When there is only one latent variable in the superstructure, our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model. We call our model the latent tree variational autoencoder (LTVAE). Whereas previous deep learning methods for clustering produce only one partition of data, LTVAE produces multiple partitions of data, each being given by one super latent variable. This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways. © 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved.