Search

Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. Specifically, most of the well-performed metrics are required to train on evaluation datasets of specific NLG tasks and evaluation dimensions, which may cause over-fitting to task-specific datasets. Furthermore, existing metrics only provide an evaluation score for each dimension without revealing the evidence to interpret how this score is obtained. To deal with these challenges, we propose a simple yet effective metric called DecompEval. This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models (PLMs) without training on evaluation datasets, aiming to enhance the generalization ability. To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result. Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability., Accepted by ACL 2023 (Main Conference)

In classification problems, the datasets are usually imbalanced, noisy or complex. Most sampling algorithms only make some improvements to the linear sampling mechanism of the synthetic minority oversampling technique (SMOTE). Nevertheless, linear oversampling has several unavoidable drawbacks. Linear oversampling is susceptible to overfitting, and the synthetic samples lack diversity and rarely account for the original distribution characteristics. An informed nonlinear oversampling framework with the granular ball (INGB) as a new direction of oversampling is proposed in this paper. It uses granular balls to simulate the spatial distribution characteristics of datasets, and informed entropy is utilized to further optimize the granular-ball space. Then, nonlinear oversampling is performed by following high-dimensional sparsity and the isotropic Gaussian distribution. Furthermore, INGB has good compatibility. Not only can it be combined with most SMOTE-based sampling algorithms to improve their performance, but it can also be easily extended to noisy imbalanced multi-classification problems. The mathematical model and theoretical proof of INGB are given in this work. Extensive experiments demonstrate that INGB outperforms the traditional linear sampling frameworks and algorithms in oversampling on complex datasets., 15 pages, 6 figures

This study proposes a multitask learning architecture for extractive summarization with coherence boosting. The architecture contains an extractive summarizer and coherent discriminator module. The coherent discriminator is trained online on the sentence vectors of the augmented textual input, thus improving its general ability of judging whether the input sentences are coherent. Meanwhile, we maximize the coherent scores from the coherent discriminator by updating the parameters of the summarizer. To make the extractive sentences trainable in a differentiable manner, we introduce two strategies, including pre-trained converting model (model-based) and converting matrix (MAT-based) that merge sentence representations. Experiments show that our proposed method significantly improves the proportion of consecutive sentences in the extracted summaries based on their positions in the original article (i.e., automatic sentence-level coherence metric), while the goodness in terms of other automatic metrics (i.e., Rouge scores and BertScores) are preserved. Human evaluation also evidences the improvement of coherence and consistency of the extracted summaries given by our method., 11 pages, 4 figures

Length-controllable machine translation is a type of constrained translation. It aims to contain the original meaning as much as possible while controlling the length of the translation. We can use automatic summarization or machine translation evaluation metrics for length-controllable machine translation, but this is not necessarily suitable and accurate. This work is the first attempt to evaluate the automatic metrics for length-controllable machine translation tasks systematically. We conduct a rigorous human evaluation on two translation directions and evaluate 18 summarization or translation evaluation metrics. We find that BLEURT and COMET have the highest correlation with human evaluation and are most suitable as evaluation metrics for length-controllable machine translation.

Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.} In this paper, we analyze the impact and utility of such matrices in the context of neural machine translation (NMT). We show that detracting syntactic and semantic information from word embeddings and running NMT systems with random embeddings is not as damaging as it initially sounds. We also show how incorporating only a limited amount of task-specific knowledge from fully-trained embeddings can boost the performance NMT systems. Our findings demonstrate that in exchange for negligible deterioration in performance, any NMT model can be run with partially random embeddings. Working with such structures means a minimal memory requirement as there is no longer need to store large embedding tables, which is a significant gain in industrial and on-device settings. We evaluated our embeddings in translating {English} into {German} and {French} and achieved a $5.3$x compression rate. Despite having a considerably smaller architecture, our models in some cases are even able to outperform state-of-the-art baselines.

Lightweight cryptography ensures cryptography applications to devices with limited resources. Low-area implementations of linear layers usually play an essential role in lightweight cryptography. The previous works have provided plenty of methods to generate low-area implementations using 2-input xor gates for various linear layers. However, it is still challenging to search for smaller implementations using two or more inputs xor gates. This paper, inspired by Banik et al., proposes a novel approach to construct a quantity of lower area implementations with (n + 1)- input gates based on the given implementations with n-input gates. Based on the novel algorithm, we present the corresponding search algorithms for n = 2 and n = 3, which means that we can efficiently convert an implementation with 2-input xor gates and 3-input xor gates to lower-area implementations with 3-input xor gates and 4-input xor gates, respectively.We improve the previous implementations of linear layers for many block ciphers according to the area with these search algorithms. For example, we achieve a better implementation with 4-input xor gates for AES MixColumns, which only requires 243 GE in the STM 130 nm library, while the previous public result is 258.9 GE. Besides, we obtain better implementations for all 5500 lightweight matrices proposed by Li et al. at FSE 2019, and the area for them is decreased by about 21% on average.

Homographs, words with the same spelling but different meanings, remain challenging in Neural Machine Translation (NMT). While recent works leverage various word embedding approaches to differentiate word sense in NMT, they do not focus on the pivotal components in resolving ambiguities of homographs in NMT: the hidden states of an encoder. In this paper, we propose a novel approach to tackle homographic issues of NMT in the latent space. We first train an encoder (aka "HDR-encoder") to learn universal sentence representations in a natural language inference (NLI) task. We further fine-tune the encoder using homograph-based synset sentences from WordNet, enabling it to learn word-level homographic disambiguation representations (HDR). The pre-trained HDR-encoder is subsequently integrated with a transformer-based NMT in various schemes to improve translation accuracy. Experiments on four translation directions demonstrate the effectiveness of the proposed method in enhancing the performance of NMT systems in the BLEU scores (up to +2.3 compared to a solid baseline). The effects can be verified by other metrics (F1, precision, and recall) of translation accuracy in an additional disambiguation task. Visualization methods like heatmaps, T-SNE and translation examples are also utilized to demonstrate the effects of the proposed method.

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-{\Sigma}. With parameter inherent from PanGu-{\alpha}, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-{\Sigma} provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.

The emerging classical-quantum transfer learning paradigm has brought a decent performance to quantum computational models in many tasks, such as computer vision, by enabling a combination of quantum models and classical pre-trained neural networks. However, using quantum computing with pre-trained models has yet to be explored in natural language processing (NLP). Due to the high linearity constraints of the underlying quantum computing infrastructures, existing Quantum NLP models are limited in performance on real tasks. We fill this gap by pre-training a sentence state with complex-valued BERT-like architecture, and adapting it to the classical-quantum transfer learning scheme for sentence classification. On quantum simulation experiments, the pre-trained representation can bring 50\% to 60\% increases to the capacity of end-to-end quantum models.

Colloids of natural river water is a key intermediate carrier of lead (Pb). It is important to monitor the transport–transformation behavior of Pb in the colloidal phase of seaward water because this behavior is related to the levels of pollution input and environmental risks posed to the sea, especially in coastal delta areas. In this study, the fractionation behavior and distribution of toxic Pb from the truly dissolved phase and the different colloidal phases in seven seaward rivers in the Yellow River Delta were investigated. The concentrations of total dissolved Pb, truly dissolved Pb, and colloidal Pb were 0.99–40.09 μg L–1, 0.40–8.10 μg L–1, and 0.60–35.88 μg L–1, respectively. In freshwater rivers, the main component of total dissolved Pb (about > 50%) is truly dissolved Pb but the main component of total dissolved Pb in the seawater environment is colloidal Pb (> 80%). A dramatic increase in salinity causes the deposition (about ≈94%) of all forms of Pb to sediment from estuarine water in winter. However, this sedimentation behavior of colloidal Pb gradually decreases (in the Shenxiangou River) when the river salinity approaches seawater salinity (S = ≈29). In the industrial port (Xiaoqinghe River) and mariculture (Yongfenghe River) estuarine areas, which have extensive seawater, the deposition behavior of colloidal Pb (

Multimodal abstractive summarization for videos (MAS) requires generating a concise textual summary to describe the highlights of a video according to multimodal resources, in our case, the video content and its transcript. Inspired by the success of the large-scale generative pre-trained language model (GPLM) in generating high-quality textual content (e.g., summary), recent MAS methods have proposed to adapt the GPLM to this task by equipping it with the visual information, which is often obtained through a general-purpose visual feature extractor. However, the generally extracted visual features may overlook some summary-worthy visual information, which impedes model performance. In this work, we propose a novel approach to learning the summary-worthy visual representation that facilitates abstractive summarization. Our method exploits the summary-worthy information from both the cross-modal transcript data and the knowledge that distills from the pseudo summary. Extensive experiments on three public multimodal datasets show that our method outperforms all competing baselines. Furthermore, with the advantages of summary-worthy visual information, our model can have a significant improvement on small datasets or even datasets with limited training data., Comment: Accepted by IJCAI-2023

Additional file 2: Table S1. Included studies on prevalence of canine dirofilariasis in China.

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-Σ. With parameter inherent from PanGu-α, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-Σ provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.

Additional file 2: Figure S1. Distribution of 17-mer depth for estimating the genome size. The x-axis represents k-mer depths while y-axis represents the proportions. The blue line represents the proportion of 17-mer in each depth. The peak depth is at 25-fold and total number of 17-mer is 61,791,522,108. The South China tiger (P. t. amoyensis) genome size was estimated to be 2471.66 Mb from the formula: Genome size = K-mer_number/K_depth of peak. Figure S2. Hi-C chromosomal contact heat map. A 500 kb resolution was used to calculate the number of Hi-C read pairs in any two bins. Figure S3. The collinearity between the South China tiger and domestic cat (Felis catus) genomes. Each dot represents an aligned region while the minimum length is 10 kb. The red dot represents forward comparison and the blue dot reverse comparison. Figure S4. Q30 and GC content for each sample. The average Q30 and GC content are 93.24% and 42.04%, respectively. Their very low variation reflected our re-sequencing data to be high quality. Figure S5. The distribution of mapping rates and average mapping depths for each individual. The depths ranged from 12.91× to 18.96 × while the rates varied from 96.03% to 98.92%. Figure S6. The distribution of the SNP number and frequency. Number of genome-wide SNPs (top) and frequency of SNPs per 1 kb of each tiger sample (bottom). Figure S7. Nucleotide diversity π estimates of six tiger subspecies. Figure S8. Heterozygosity statistics of genome-wide SNPs. (a) Observed heterozygosity across all individual genomes of each tiger subspecies. (b) Genomic heterozygosity in each tiger subspecies at population level. Figure S9. Pairwise FST values between six tiger subspecies. The weighted FST values are shown above the diagonal while their standard deviations below the diagonal. Figure S10. Principal component analysis using genome-wide SNPs of six tiger subspecies. Figure S11. A maximum likelihood tree was built using TreeMix software with whole-genome sequencing data of six tiger subspecies and cat. Figure S12. Cluster analysis based on the matrix of genome-wide identity-by-state (IBS) pairwise distances between six tiger subspecies determined by a permutation score. Figure S13. Analysis of genome-wide average identity-by-state (IBS) pairwise identities between six tiger subspecies. Figure S14. Population genetic structure of the South China tigers estimated by the ADMIXTURE. Figure S15. Demographic histories of tiger subspecies. PSMC analysis shows the change in effective population size over time. The dash lines represent 100 bootstraps. The representative individuals sequenced at a high read coverage were selected for each graph, South China tiger (ptam_4, mean 25.38×), Amur tiger (ptal_1, mean 14.76×), Indochinese tiger (ptco_2, mean 13.98×), Malayan tiger (ptja_7, mean 13.13×), Bengal tiger (ptti_3, mean 13.79×), and Sumatran tiger (ptsu_1, mean 13.41×). Figure S16. The results of Dsuite. Heatmap showing statistical support for introgression between pairs of tiger subspecies. Cells in the heatmap indicate the pairwise Z score values between the branch b identified on the expanded tree on the Y axis (relative to its sister branch) and the taxa P3 identified on the X-axis. The grey color indicates the none. (a) Result of branch for South China tiger (lineage 1) with the other tiger subspecies populations. (b) Result of branch for South China tiger (lineage 2) with the other tiger subspecies populations. (c) Result of branch for South China tiger (ptam1 individual) with the other tiger subspecies populations. Figure S17. The graph of ABBA-BABA test. The values of D (pop1, pop2; pop3, pop4) >0 indicates that there are gene flows between pop1 and pop3. Here, only the introgression occurred in South China tiger will be shown. (a) D-statistic for South China tiger (lineage 1) with other tiger subspecies. (b) D-statistic for South China tiger (lineage 2) with other tiger subspecies. (c) D-statistic for South China tiger (ptam1 individual) with other tiger subspecies. Figure S18. Plot of inferred introgression between the South China tiger lineage 1 and other tiger subspecies populations detected by the TreeMix method. The scale bar shows 10 times the average standard error of the entries in the sample covariance matrix. Figure S19. Plot of inferred introgression between the South China tiger lineage 2 and other tiger subspecies populations detected by the TreeMix method. The scale bar shows 10 times the average standard error of the entries in the sample covariance matrix. Figure S20. Plot of inferred introgression between the South China tiger ptam_1 individual and other tiger subspecies populations detected by the TreeMix method. The scale bar shows 10 times the average standard error of the entries in the sample covariance matrix. Figure S21. Genomic inbreeding coefficients FH in each tiger subspecies. Figure S22. The plot of three inbreeding coefficients (FP, FH, and FROH) of the South China tiger. Figure S23. Comparison of homozygote and heterozygote percentage of per mutation category among six tiger subspecies. The significant value was calculated by t-test.

Additional file 4: Preferred reporting items for systematic reviews and meta-analyses checklist items.

Large language models have recently made tremendous progress in a variety of aspects, e.g., cross-task generalization, instruction following. Comprehensively evaluating the capability of large language models in multiple tasks is of great importance. In this paper, we propose M3KE, a Massive Multi-Level Multi-Subject Knowledge Evaluation benchmark, which is developed to measure knowledge acquired by Chinese large language models by testing their multitask accuracy in zero- and few-shot settings. We have collected 20,477 questions from 71 tasks. Our selection covers all major levels of Chinese education system, ranging from the primary school to college, as well as a wide variety of subjects, including humanities, history, politics, law, education, psychology, science, technology, art and religion. All questions are multiple-choice questions with four options, hence guaranteeing a standardized and unified assessment process. We've assessed a number of state-of-the-art open-source Chinese large language models on the proposed benchmark. The size of these models varies from 335M to 130B parameters. Experiment results demonstrate that they perform significantly worse than GPT-3.5 that reaches an accuracy of ~ 48% on M3KE. The dataset is available at https://github.com/tjunlp-lab/M3KE.

Additional file 4: Preferred reporting items for systematic reviews and meta-analyses checklist items.

Additional file 2: Fig. s1. The genetic linkage map of Hi-C data. Fig. s2. Phylogenetic analyses based on ML and Bayesian inference methods. Fig. s3-5. Heat map of homologous genes in pairwise chromosomes of A. fangsiao and O. sinensis and N. pompilius. Fig. s6. Synteny analyses between A. fangsiao, O. sinensis, and N. pompilius. Fig. s7. Genomic organization of protocadherin, GPCR and C2H2 ZNF in A. fangsiao and O. sinensis. Fig. s8. Divergent time of C2H2 ZNF. Fig. s9. Cell population composition of the supra-esophageal brain. Fig. s10. Expression of top 10 marker genes of cell type 1-8. Fig. s11. Comparisons of functions in cell type 2-8. Fig. s12. Bulk transcriptomic expression of 10 marker genes. Fig. s13. Bar plot reflected the number of C2H2 ZNF (top), protocadherin (middle), and GPCR (bottom) genes in marker genes of cell type 1-8. Fig. s14. Heatmap of GPCR, C2H2 ZNF, and protocadherin (x axis) expressions in the sup-brain cell types (x axis). Fig. s15. Boxplot of average expression of protocadherin (a), GPCR (b), and C2H2 ZNF (c) in cells.

Utilizing pivot language effectively can significantly improve low-resource machine translation. Usually, the two translation models, source-pivot and pivot-target, are trained individually and do not utilize the limited (source, target) parallel data. This work proposes an end-to-end training method for the cascaded translation model and configures an improved decoding algorithm. The input of the pivot-target model is modified to weighted pivot embedding based on the probability distribution output by the source-pivot model. This allows the model to be trained end-to-end. In addition, we mitigate the inconsistency between tokens and probability distributions while using beam search in pivot decoding. Experiments demonstrate that our method enhances the quality of translation.

To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.

Disentangled representation learning remains challenging as ground truth factors of variation do not naturally exist. To address this, we present Vocabulary Disentanglement Retrieval~(VDR), a simple yet effective retrieval-based disentanglement framework that leverages nature language as distant supervision. Our approach is built upon the widely-used bi-encoder architecture with disentanglement heads and is trained on data-text pairs that are readily available on the web or in existing datasets. This makes our approach task- and modality-agnostic with potential for a wide range of downstream applications. We conduct experiments on 16 datasets in both text-to-text and cross-modal scenarios and evaluate VDR in a zero-shot setting. With the incorporation of disentanglement heads and a minor increase in parameters, VDR achieves significant improvements over the base retriever it is built upon, with a 9% higher on NDCG@10 scores in zero-shot text-to-text retrieval and an average of 13% higher recall in cross-modal retrieval. In comparison to other baselines, VDR outperforms them in most tasks, while also improving explainability and efficiency.

Previous works show that Pre-trained Language Models (PLMs) can capture factual knowledge. However, some analyses reveal that PLMs fail to perform it robustly, e.g., being sensitive to the changes of prompts when extracting factual knowledge. To mitigate this issue, we propose to let PLMs learn the deterministic relationship between the remaining context and the masked content. The deterministic relationship ensures that the masked factual content can be deterministically inferable based on the existing clues in the context. That would provide more stable patterns for PLMs to capture factual knowledge than randomly masking. Two pre-training tasks are further introduced to motivate PLMs to rely on the deterministic relationship when filling masks. Specifically, we use an external Knowledge Base (KB) to identify deterministic relationships and continuously pre-train PLMs with the proposed methods. The factual knowledge probing experiments indicate that the continuously pre-trained PLMs achieve better robustness in factual knowledge capturing. Further experiments on question-answering datasets show that trying to learn a deterministic relationship with the proposed methods can also help other knowledge-intensive tasks., Accepted at EMNLP 2022

MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) is a multilingual dataset we have built for the WSDM 2023 Cup challenge that focuses on ad hoc retrieval across 18 different languages, which collectively encompass over three billion native speakers around the world. These languages have diverse typologies, originate from many different language families, and are associated with varying amounts of available resources -- including what researchers typically characterize as high-resource as well as low-resource languages. Our dataset is designed to support the creation and evaluation of models for monolingual retrieval, where the queries and the corpora are in the same language. In total, we have gathered over 700k high-quality relevance judgments for around 77k queries over Wikipedia in these 18 languages, where all assessments have been performed by native speakers hired by our team. Our goal is to spur research that will improve retrieval across a continuum of languages, thus enhancing information access capabilities for diverse populations around the world, particularly those that have been traditionally underserved. This overview paper describes the dataset and baselines that we share with the community. The MIRACL website is live at http://miracl.ai/.

Collecting supporting evidence from large corpora of text (e.g., Wikipedia) is of great challenge for open-domain Question Answering (QA). Especially, for multi-hop open-domain QA, scattered evidence pieces are required to be gathered together to support the answer extraction. In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering. Specifically, the hop in this paper is defined as the combination of a hyperlink and the corresponding outbound link document. The hyperlink is encoded as the mention embedding which models the structured knowledge of how the outbound link entity is mentioned in the textual context, and the corresponding outbound link document is encoded as the document embedding representing the unstructured knowledge within it. Accordingly, we build HopRetriever which retrieves hops over Wikipedia to answer complex questions. Experiments on the HotpotQA dataset demonstrate that HopRetriever outperforms previously published evidence retrieval methods by large margins. Moreover, our approach also yields quantifiable interpretations of the evidence collection process., Accepted at AAAI 2021

Aimed at the problem that the traditional meta-path random walk in heterogeneous network representation cannot accurately describe the heterogeneous network structure and cannot capture the true distribution of network nodes well, a heterogeneous network representation method based on variational inference and meta-path decomposition is proposed, which is named HetVAE. First, combining with the idea of path similarity, a node selection strategy is designed to improve the random walk of the meta-path. Next, the variational theory is introduced to effectively sample the latent variables in the original distribution. After that, a personalized attention machanism is implemented, which weights the node vector representation of different sub-networks obtained by decomposition. Then, these node vectors are fused by the proposed model, so that the final node vector representation can have richer semantic information. Finally, several experiments on different network tasks are performed on the three real data sets of DBLP, AMiner, and Yelp. The effectiveness of the model is verified by these results. In node classification and node clustering tasks, compared with some state-of-the-art algorithms, the Micro-F1 and normalized mutual information (NMI) increase by 1.12% to 4.36% and 1.35% to 18% respectively. It is proved that HetVAE can effectively capture the heterogeneous network structure and learn the node vetcor representation that conforms more with the true distribution.

We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem description. We train PanGu-Coder using a two-stage strategy: the first stage employs Causal Language Modelling (CLM) to pre-train on raw programming language data, while the second stage uses a combination of Causal Language Modelling and Masked Language Modelling (MLM) training objectives that focus on the downstream task of text-to-code generation and train on loosely curated pairs of natural language program definitions and code functions. Finally, we discuss PanGu-Coder-FT, which is fine-tuned on a combination of competitive programming problems and code with continuous integration tests. We evaluate PanGu-Coder with a focus on whether it generates functionally correct programs and demonstrate that it achieves equivalent or better performance than similarly sized models, such as CodeX, while attending a smaller context window and training on less data., 27 pages

Pinyin to Character conversion (P2C) task is the key task of Input Method Engine (IME) in commercial input software for Asian languages, such as Chinese, Japanese, Thai language and so on. It's usually treated as sequence labelling task and resolved by language model, i.e. n-gram or RNN. However, the low capacity of the n-gram or RNN limits its performance. This paper introduces a new solution named PERT which stands for bidirectional Pinyin Encoder Representations from Transformers. It achieves significant improvement of performance over baselines. Furthermore, we combine PERT with n-gram under a Markov framework, and improve performance further. Lastly, the external lexicon is incorporated into PERT so as to resolve the OOD issue of IME.

Objective To explore the psychological experience and needs of female caregivers of stroke patients in rehabilitation period. Methods The phenomenological research method was used to conduct semi-structured interviews with 12 female caregivers of stroke patients. Results Based on the real experience of female caregivers during stroke rehabilitation period, four themes were extracted, including imbalance in physical and mental health, coping with multiple roles, insufficient of nursing information acquisition and lack of social support. Conclusion The feelings of female caregivers of patients in stroke rehabilitation stage are multi-dimensional, suggesting that we should pay attention to the mental health of female caregivers and guide them to seek positive coping styles, so as to provide better care for the rehabilitation of stroke patients.

The article examines the contradictions that caused the need for digital support for the process of learning the musical culture of China by future music teachers of Russia. In search of approaches to the development of such software, the authors come to the idea of the expediency of creating an electronic educational resource “Musical Culture of China”. The methodological basis for determining its content and structure was the musicological, psychological and pedagogical approaches. In their entirety, they provide a multi­vector characteristic of the identity of Chinese musical culture, its national characteristics. This resource has two levels, basic and advanced. When describing each of them, the justification of their block­ modular construction and the pedagogical purpose of all the selected structural components are given. At the basic level, students get acquainted with the predominantly knowledge component. The content of the advanced level is aimed at expanding the acquired knowledge and their practical application. The work with this electronic educational resource helps to increase the readiness of future teachers to implement the acquired knowledge and practical skills in their professional activities when introducing students to Chinese musical culture in musical­ instrumental, musical­plastic and vocal­choral activities.

BackgroundPrevious studies have suggested that pregnant women with pulmonary hypertension (PH) have high maternal mortality. However, indexes or factors that can predict maternal death are lacking.MethodsWe retrospectively reviewed pregnant women with PH admitted for delivery from 2012 to 2020 and followed them for over 6 months. The patients were divided into two groups according to 10-day survival status after delivery. Predictive models and predictors for maternal death were identified using four machine learning algorithms: naïve Bayes, random forest, gradient boosting decision tree (GBDT), and support vector machine.ResultsA total of 299 patients were included. The most frequent PH classifications were Group 1 PH (73.9%) and Group 2 PH (23.7%). The mortality within 10 days after delivery was 9.4% and higher in Group 1 PH than in the other PH groups (11.7 vs. 2.6%, P = 0.016). We identified 17 predictors, each with a P-value < 0.05 by univariable analysis, that were associated with an increased risk of death, and the most notable were pulmonary artery systolic pressure (PASP), platelet count, red cell distribution width, N-terminal brain natriuretic peptide (NT-proBNP), and albumin (all P < 0.01). Four prediction models were established using the candidate variables, and the GBDT model showed the best performance (F1-score = 66.7%, area under the curve = 0.93). Feature importance showed that the three most important predictors were NT-proBNP, PASP, and albumin.ConclusionMortality remained high, particularly in Group 1 PH. Our study shows that NT-proBNP, PASP, and albumin are the most important predictors of maternal death in the GBDT model. These findings may help clinicians provide better advice regarding fertility for women with PH.

Triangular machine translation is a special case of low-resource machine translation where the language pair of interest has limited parallel data, but both languages have abundant parallel data with a pivot language. Naturally, the key to triangular machine translation is the successful exploitation of such auxiliary data. In this work, we propose a transfer-learning-based approach that utilizes all types of auxiliary data. As we train auxiliary source-pivot and pivot-target translation models, we initialize some parameters of the pivot side with a pre-trained language model and freeze them to encourage both translation models to work in the same pivot language space, so that they can be smoothly transferred to the source-target translation model. Experiments show that our approach can outperform previous ones., ACL 2022

The research of open-domain dialog systems has been greatly prospered by neural models trained on large-scale corpora, however, such corpora often introduce various safety problems (e.g., offensive languages, biases, and toxic behaviors) that significantly hinder the deployment of dialog systems in practice. Among all these unsafe issues, addressing social bias is more complex as its negative impact on marginalized populations is usually expressed implicitly, thus requiring normative reasoning and rigorous analysis. In this paper, we focus our investigation on social bias detection of dialog safety problems. We first propose a novel Dial-Bias Frame for analyzing the social bias in conversations pragmatically, which considers more comprehensive bias-related analyses rather than simple dichotomy annotations. Based on the proposed framework, we further introduce CDail-Bias Dataset that, to our knowledge, is the first well-annotated Chinese social bias dialog dataset. In addition, we establish several dialog bias detection benchmarks at different label granularities and input types (utterance-level and context-level). We show that the proposed in-depth analyses together with these benchmarks in our Dial-Bias Frame are necessary and essential to bias detection tasks and can benefit building safe dialog systems in practice.

Background and Purpose Neuropathic pain is a major side-effect of paclitaxel (PTX) chemotherapy. Although the precise mechanisms responsible for this pain are unclear, the activation of neuroglia and upregulation of the TLR4/NF-κB pathway are known to be involved. In this study, we determined whether electroacupuncture (EA) could limit mechanical hypersensitivity resulting from the chemotherapeutic drug PTX in rats, and investigated the potential mechanisms involved. Methods Rats intraperitoneally received a cumulative dose of 8 mg/kg PTX (2 mg/kg per day) or vehicle control on alternate days (day 0, 2, 4 and 6). EA treatment (10 Hz, 1 mA) was applied at bilateral ST36 acupoints in rats once every other day on days 0–14. For sham EA, needles were inserted at ST36 acupoints without electrical stimulation. Mechanical allodynia was measured by mechanical withdrawal latency (MWL) of paws to a mechanical stimulus every 2 days. Protein expression of TLR4 and NF-κB p65, as well as TMEM119 and GFAP (indicators of microglia and astrocytes, respectively) in spinal cord was quantified by Western blot analysis. Levels of inflammatory cytokines IL-1β and TNF-α in spinal cord and serum were detected by ELISA. Results Mechanical allodynia induced by PTX in both paws (right and left) of rats was significantly attenuated by EA but not sham EA treatment. In addition, EA, but not sham EA, inhibited the activation of both microglia (TMEM119) and astrocytes (GFAP) in lumbar spinal cord. Moreover, Western blot analysis revealed that protein expression of TLR4 and NF-κB in spinal cord was suppressed by EA but not sham EA treatment. PTX significantly increased inflammatory cytokines in spinal cord and serum, which were ameliorated by EA treatment but not by sham EA. Conclusion These results indicate that EA treatment attenuates PTX-induced mechanical allodynia. The putative mechanism corroborating this finding could be related to the suppression of activated microglia and astrocytes in spinal cord, as well as the inhibition of the activated TLR4/NF-κB signaling pathway by EA treatment.

Real human conversation data are complicated, heterogeneous, and noisy, from which building open-domain dialogue systems remains a challenging task. In fact, such dialogue data still contains a wealth of information and knowledge, however, they are not fully explored. In this paper, we show existing open-domain dialogue generation methods that memorize context-response paired data with autoregressive or encode-decode language models underutilize the training data. Different from current approaches, using external knowledge, we explore a retrieval-generation training framework that can take advantage of the heterogeneous and noisy training data by considering them as "evidence". In particular, we use BERTScore for retrieval, which gives better qualities of the evidence and generation. Experiments over publicly available datasets demonstrate that our method can help models generate better responses, even such training data are usually impressed as low-quality data. Such performance gain is comparable with those improved by enlarging the training set, even better. We also found that the model performance has a positive correlation with the relevance of the retrieved evidence. Moreover, our method performed well on zero-shot experiments, which indicates that our method can be more robust to real-world data., Accepted in COLING 2022

Although lyrics generation has achieved significant progress in recent years, it has limited practical applications because the generated lyrics cannot be performed without composing compatible melodies. In this work, we bridge this practical gap by proposing a song rewriting system which rewrites the lyrics of an existing song such that the generated lyrics are compatible with the rhythm of the existing melody and thus singable. In particular, we propose SongRewriter,a controllable Chinese lyrics generation and editing system which assists users without prior knowledge of melody composition. The system is trained by a randomized multi-level masking strategy which produces a unified model for generating entirely new lyrics or editing a few fragments. To improve the controllabiliy of the generation process, we further incorporate a keyword prompt to control the lexical choices of the content and propose novel decoding constraints and a vowel modeling task to enable flexible end and internal rhyme schemes. While prior rhyming metrics are mainly for rap lyrics, we propose three novel rhyming evaluation metrics for song lyrics. Both automatic and human evaluations show that the proposed model performs better than the state-of-the-art models in both contents and rhyming quality., Comment: ACL Findings 2023

There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language. This work concerns addressing two major problems in existing Arabic PLMs which constraint progress of the Arabic NLU and NLG fields.First, existing Arabic PLMs are not well-explored and their pre-trainig can be improved significantly using a more methodical approach. Second, there is a lack of systematic and reproducible evaluation of these models in the literature. In this work, we revisit both the pre-training and evaluation of Arabic PLMs. In terms of pre-training, we explore improving Arabic LMs from three perspectives: quality of the pre-training data, size of the model, and incorporating character-level information. As a result, we release three new Arabic BERT-style models ( JABER, Char-JABER, and SABER), and two T5-style models (AT5S and AT5B). In terms of evaluation, we conduct a comprehensive empirical study to systematically evaluate the performance of existing state-of-the-art models on ALUE that is a leaderboard-powered benchmark for Arabic NLU tasks, and on a subset of the ARGEN benchmark for Arabic NLG tasks. We show that our models significantly outperform existing Arabic PLMs and achieve a new state-of-the-art performance on discriminative and generative Arabic NLU and NLG tasks. Our models and source code to reproduce of results will be made available shortly.

Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive due to the requirement of millions of video-text pairs and the redundant data structure of each video. To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training. To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also propose a non-parametric pooling mechanism to adaptively reweight the fine-grained video embedding conditioned on the text. Experimental results on text-video retrieval and video question answering show that the proposed LiteVL even outperforms previous video-language pre-trained models by a clear margin, though without any video-language pre-training., Comment: 13 pages, 6 figures, accepted by EMNLP 2022 main conference

Additional file 2: Figure S1. a protein sequence alignment of TgANT using MEGA7 from Toxoplasma gondii, Saccharomyces cerevisiae, Plasmodium falciparum, Cyclospora cayetanensis, Arabidopsis thaliana, Cardiosporidium cionae, Homo sapiens and other species. b Prediction results of TMHMM Server v.2.0 showed that TgANT has four transmembrane domains.

Recently, semantic parsing using hierarchical representations for dialog systems has captured substantial attention. Task-Oriented Parse (TOP), a tree representation with intents and slots as labels of nested tree nodes, has been proposed for parsing user utterances. Previous TOP parsing methods are limited on tackling unseen dynamic slot values (e.g., new songs and locations added), which is an urgent matter for real dialog systems. To mitigate this issue, we first propose a novel span-splitting representation for span-based parser that outperforms existing methods. Then we present a novel lexicon-injected semantic parser, which collects slot labels of tree representation as a lexicon, and injects lexical features to the span representation of parser. An additional slot disambiguation technique is involved to remove inappropriate span match occurrences from the lexicon. Our best parser produces a new state-of-the-art result (87.62%) on the TOP dataset, and demonstrates its adaptability to frequently updated slot lexicon entries in real task-oriented dialog, with no need of retraining.

Morality in dialogue systems has raised great attention in research recently. A moral dialogue system aligned with users' values could enhance conversation engagement and user connections. In this paper, we propose a framework, MoralDial to train and evaluate moral dialogue systems. In our framework, we first explore the communication mechanisms of morality and resolve expressed morality into three parts, which indicate the roadmap for building a moral dialogue system. Based on that, we design a simple yet effective method: constructing moral discussions between simulated specific users and the dialogue system. The constructed discussions consist of expressing, explaining, revising, and inferring moral views in dialogue exchanges, which makes conversational models learn morality well in a natural manner. Furthermore, we propose a novel evaluation method under the framework. We evaluate the multiple aspects of morality by judging the relation between dialogue responses and human values in discussions, where the multifaceted nature of morality is particularly considered. Automatic and manual experiments demonstrate that our framework is promising to train and evaluate moral dialogue systems., Comment: Accepted to ACL 2023

Supplemental material, sj-docx-1-onc-10.1177_11795549221116834 for Concordance of Genomic Profiles in Matched Tissue and Plasma Samples From Chinese Patients With Lung Cancer by Yueming He, Weifeng Guo, Meng Xu, Junling Huang, Xiange Zhang, Huanzhang Su, Dongxia Hong and Qun Liu in Clinical Medicine Insights: Oncology

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with $7\times$ fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks., Accepted to ACL 2022

Additional file 1: Table S1. Nucleotide sequences of primers used for RT-qPCR.

Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding~(VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that our Wukong-Reader has superior performance on various VDU tasks such as information extraction. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.

Recent work explored the potential of large-scale Transformer-based pre-trained models, especially Pre-trained Language Models (PLMs) in natural language processing. This raises many concerns from various perspectives, e.g., financial costs and carbon emissions. Compressing PLMs like BERT with negligible performance loss for faster inference and cheaper deployment has attracted much attention. In this work, we aim to explore larger compression ratios for PLMs, among which tensor decomposition is a potential but under-investigated one. Two decomposition and reconstruction protocols are further proposed to improve the effectiveness and efficiency during compression. Our compressed BERT with ${1}/{7}$ parameters in Transformer layers performs on-par with, sometimes slightly better than the original BERT in GLUE benchmark. A tiny version achieves $96.7\%$ performance of BERT-base with $ {1}/{48} $ encoder parameters (i.e., less than 2M parameters excluding the embedding layer) and $2.7 \times$ faster on inference. To show that the proposed method is orthogonal to existing compression methods like knowledge distillation, we also explore the benefit of the proposed method on a distilled BERT., Comment: Accepted at ICLR2022. Code available at https://github.com/twinkle0331/Xcompression

Supplemental material, sj-docx-1-onc-10.1177_11795549221116834 for Concordance of Genomic Profiles in Matched Tissue and Plasma Samples From Chinese Patients With Lung Cancer by Yueming He, Weifeng Guo, Meng Xu, Junling Huang, Xiange Zhang, Huanzhang Su, Dongxia Hong and Qun Liu in Clinical Medicine Insights: Oncology