Search

Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this paper words and subwords are referred to as \textit{tokens} and the term \textit{embedding} only refers to embeddings of inputs.} In this paper, we analyze the impact and utility of such matrices in the context of neural machine translation (NMT). We show that detracting syntactic and semantic information from word embeddings and running NMT systems with random embeddings is not as damaging as it initially sounds. We also show how incorporating only a limited amount of task-specific knowledge from fully-trained embeddings can boost the performance NMT systems. Our findings demonstrate that in exchange for negligible deterioration in performance, any NMT model can be run with partially random embeddings. Working with such structures means a minimal memory requirement as there is no longer need to store large embedding tables, which is a significant gain in industrial and on-device settings. We evaluated our embeddings in translating {English} into {German} and {French} and achieved a $5.3$x compression rate. Despite having a considerably smaller architecture, our models in some cases are even able to outperform state-of-the-art baselines.

Lightweight cryptography ensures cryptography applications to devices with limited resources. Low-area implementations of linear layers usually play an essential role in lightweight cryptography. The previous works have provided plenty of methods to generate low-area implementations using 2-input xor gates for various linear layers. However, it is still challenging to search for smaller implementations using two or more inputs xor gates. This paper, inspired by Banik et al., proposes a novel approach to construct a quantity of lower area implementations with (n + 1)- input gates based on the given implementations with n-input gates. Based on the novel algorithm, we present the corresponding search algorithms for n = 2 and n = 3, which means that we can efficiently convert an implementation with 2-input xor gates and 3-input xor gates to lower-area implementations with 3-input xor gates and 4-input xor gates, respectively.We improve the previous implementations of linear layers for many block ciphers according to the area with these search algorithms. For example, we achieve a better implementation with 4-input xor gates for AES MixColumns, which only requires 243 GE in the STM 130 nm library, while the previous public result is 258.9 GE. Besides, we obtain better implementations for all 5500 lightweight matrices proposed by Li et al. at FSE 2019, and the area for them is decreased by about 21% on average.

Colloids of natural river water is a key intermediate carrier of lead (Pb). It is important to monitor the transport–transformation behavior of Pb in the colloidal phase of seaward water because this behavior is related to the levels of pollution input and environmental risks posed to the sea, especially in coastal delta areas. In this study, the fractionation behavior and distribution of toxic Pb from the truly dissolved phase and the different colloidal phases in seven seaward rivers in the Yellow River Delta were investigated. The concentrations of total dissolved Pb, truly dissolved Pb, and colloidal Pb were 0.99–40.09 μg L–1, 0.40–8.10 μg L–1, and 0.60–35.88 μg L–1, respectively. In freshwater rivers, the main component of total dissolved Pb (about > 50%) is truly dissolved Pb but the main component of total dissolved Pb in the seawater environment is colloidal Pb (> 80%). A dramatic increase in salinity causes the deposition (about ≈94%) of all forms of Pb to sediment from estuarine water in winter. However, this sedimentation behavior of colloidal Pb gradually decreases (in the Shenxiangou River) when the river salinity approaches seawater salinity (S = ≈29). In the industrial port (Xiaoqinghe River) and mariculture (Yongfenghe River) estuarine areas, which have extensive seawater, the deposition behavior of colloidal Pb (

Multimodal abstractive summarization for videos (MAS) requires generating a concise textual summary to describe the highlights of a video according to multimodal resources, in our case, the video content and its transcript. Inspired by the success of the large-scale generative pre-trained language model (GPLM) in generating high-quality textual content (e.g., summary), recent MAS methods have proposed to adapt the GPLM to this task by equipping it with the visual information, which is often obtained through a general-purpose visual feature extractor. However, the generally extracted visual features may overlook some summary-worthy visual information, which impedes model performance. In this work, we propose a novel approach to learning the summary-worthy visual representation that facilitates abstractive summarization. Our method exploits the summary-worthy information from both the cross-modal transcript data and the knowledge that distills from the pseudo summary. Extensive experiments on three public multimodal datasets show that our method outperforms all competing baselines. Furthermore, with the advantages of summary-worthy visual information, our model can have a significant improvement on small datasets or even datasets with limited training data., Comment: Accepted by IJCAI-2023

Additional file 2: Table S1. Included studies on prevalence of canine dirofilariasis in China.

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-Σ. With parameter inherent from PanGu-α, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-Σ provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.

Additional file 2: Figure S1. Distribution of 17-mer depth for estimating the genome size. The x-axis represents k-mer depths while y-axis represents the proportions. The blue line represents the proportion of 17-mer in each depth. The peak depth is at 25-fold and total number of 17-mer is 61,791,522,108. The South China tiger (P. t. amoyensis) genome size was estimated to be 2471.66 Mb from the formula: Genome size = K-mer_number/K_depth of peak. Figure S2. Hi-C chromosomal contact heat map. A 500 kb resolution was used to calculate the number of Hi-C read pairs in any two bins. Figure S3. The collinearity between the South China tiger and domestic cat (Felis catus) genomes. Each dot represents an aligned region while the minimum length is 10 kb. The red dot represents forward comparison and the blue dot reverse comparison. Figure S4. Q30 and GC content for each sample. The average Q30 and GC content are 93.24% and 42.04%, respectively. Their very low variation reflected our re-sequencing data to be high quality. Figure S5. The distribution of mapping rates and average mapping depths for each individual. The depths ranged from 12.91× to 18.96 × while the rates varied from 96.03% to 98.92%. Figure S6. The distribution of the SNP number and frequency. Number of genome-wide SNPs (top) and frequency of SNPs per 1 kb of each tiger sample (bottom). Figure S7. Nucleotide diversity π estimates of six tiger subspecies. Figure S8. Heterozygosity statistics of genome-wide SNPs. (a) Observed heterozygosity across all individual genomes of each tiger subspecies. (b) Genomic heterozygosity in each tiger subspecies at population level. Figure S9. Pairwise FST values between six tiger subspecies. The weighted FST values are shown above the diagonal while their standard deviations below the diagonal. Figure S10. Principal component analysis using genome-wide SNPs of six tiger subspecies. Figure S11. A maximum likelihood tree was built using TreeMix software with whole-genome sequencing data of six tiger subspecies and cat. Figure S12. Cluster analysis based on the matrix of genome-wide identity-by-state (IBS) pairwise distances between six tiger subspecies determined by a permutation score. Figure S13. Analysis of genome-wide average identity-by-state (IBS) pairwise identities between six tiger subspecies. Figure S14. Population genetic structure of the South China tigers estimated by the ADMIXTURE. Figure S15. Demographic histories of tiger subspecies. PSMC analysis shows the change in effective population size over time. The dash lines represent 100 bootstraps. The representative individuals sequenced at a high read coverage were selected for each graph, South China tiger (ptam_4, mean 25.38×), Amur tiger (ptal_1, mean 14.76×), Indochinese tiger (ptco_2, mean 13.98×), Malayan tiger (ptja_7, mean 13.13×), Bengal tiger (ptti_3, mean 13.79×), and Sumatran tiger (ptsu_1, mean 13.41×). Figure S16. The results of Dsuite. Heatmap showing statistical support for introgression between pairs of tiger subspecies. Cells in the heatmap indicate the pairwise Z score values between the branch b identified on the expanded tree on the Y axis (relative to its sister branch) and the taxa P3 identified on the X-axis. The grey color indicates the none. (a) Result of branch for South China tiger (lineage 1) with the other tiger subspecies populations. (b) Result of branch for South China tiger (lineage 2) with the other tiger subspecies populations. (c) Result of branch for South China tiger (ptam1 individual) with the other tiger subspecies populations. Figure S17. The graph of ABBA-BABA test. The values of D (pop1, pop2; pop3, pop4) >0 indicates that there are gene flows between pop1 and pop3. Here, only the introgression occurred in South China tiger will be shown. (a) D-statistic for South China tiger (lineage 1) with other tiger subspecies. (b) D-statistic for South China tiger (lineage 2) with other tiger subspecies. (c) D-statistic for South China tiger (ptam1 individual) with other tiger subspecies. Figure S18. Plot of inferred introgression between the South China tiger lineage 1 and other tiger subspecies populations detected by the TreeMix method. The scale bar shows 10 times the average standard error of the entries in the sample covariance matrix. Figure S19. Plot of inferred introgression between the South China tiger lineage 2 and other tiger subspecies populations detected by the TreeMix method. The scale bar shows 10 times the average standard error of the entries in the sample covariance matrix. Figure S20. Plot of inferred introgression between the South China tiger ptam_1 individual and other tiger subspecies populations detected by the TreeMix method. The scale bar shows 10 times the average standard error of the entries in the sample covariance matrix. Figure S21. Genomic inbreeding coefficients FH in each tiger subspecies. Figure S22. The plot of three inbreeding coefficients (FP, FH, and FROH) of the South China tiger. Figure S23. Comparison of homozygote and heterozygote percentage of per mutation category among six tiger subspecies. The significant value was calculated by t-test.

Additional file 4: Preferred reporting items for systematic reviews and meta-analyses checklist items.

Large language models have recently made tremendous progress in a variety of aspects, e.g., cross-task generalization, instruction following. Comprehensively evaluating the capability of large language models in multiple tasks is of great importance. In this paper, we propose M3KE, a Massive Multi-Level Multi-Subject Knowledge Evaluation benchmark, which is developed to measure knowledge acquired by Chinese large language models by testing their multitask accuracy in zero- and few-shot settings. We have collected 20,477 questions from 71 tasks. Our selection covers all major levels of Chinese education system, ranging from the primary school to college, as well as a wide variety of subjects, including humanities, history, politics, law, education, psychology, science, technology, art and religion. All questions are multiple-choice questions with four options, hence guaranteeing a standardized and unified assessment process. We've assessed a number of state-of-the-art open-source Chinese large language models on the proposed benchmark. The size of these models varies from 335M to 130B parameters. Experiment results demonstrate that they perform significantly worse than GPT-3.5 that reaches an accuracy of ~ 48% on M3KE. The dataset is available at https://github.com/tjunlp-lab/M3KE.

Additional file 4: Preferred reporting items for systematic reviews and meta-analyses checklist items.

Additional file 2: Fig. s1. The genetic linkage map of Hi-C data. Fig. s2. Phylogenetic analyses based on ML and Bayesian inference methods. Fig. s3-5. Heat map of homologous genes in pairwise chromosomes of A. fangsiao and O. sinensis and N. pompilius. Fig. s6. Synteny analyses between A. fangsiao, O. sinensis, and N. pompilius. Fig. s7. Genomic organization of protocadherin, GPCR and C2H2 ZNF in A. fangsiao and O. sinensis. Fig. s8. Divergent time of C2H2 ZNF. Fig. s9. Cell population composition of the supra-esophageal brain. Fig. s10. Expression of top 10 marker genes of cell type 1-8. Fig. s11. Comparisons of functions in cell type 2-8. Fig. s12. Bulk transcriptomic expression of 10 marker genes. Fig. s13. Bar plot reflected the number of C2H2 ZNF (top), protocadherin (middle), and GPCR (bottom) genes in marker genes of cell type 1-8. Fig. s14. Heatmap of GPCR, C2H2 ZNF, and protocadherin (x axis) expressions in the sup-brain cell types (x axis). Fig. s15. Boxplot of average expression of protocadherin (a), GPCR (b), and C2H2 ZNF (c) in cells.

Utilizing pivot language effectively can significantly improve low-resource machine translation. Usually, the two translation models, source-pivot and pivot-target, are trained individually and do not utilize the limited (source, target) parallel data. This work proposes an end-to-end training method for the cascaded translation model and configures an improved decoding algorithm. The input of the pivot-target model is modified to weighted pivot embedding based on the probability distribution output by the source-pivot model. This allows the model to be trained end-to-end. In addition, we mitigate the inconsistency between tokens and probability distributions while using beam search in pivot decoding. Experiments demonstrate that our method enhances the quality of translation.

Collecting supporting evidence from large corpora of text (e.g., Wikipedia) is of great challenge for open-domain Question Answering (QA). Especially, for multi-hop open-domain QA, scattered evidence pieces are required to be gathered together to support the answer extraction. In this paper, we propose a new retrieval target, hop, to collect the hidden reasoning evidence from Wikipedia for complex question answering. Specifically, the hop in this paper is defined as the combination of a hyperlink and the corresponding outbound link document. The hyperlink is encoded as the mention embedding which models the structured knowledge of how the outbound link entity is mentioned in the textual context, and the corresponding outbound link document is encoded as the document embedding representing the unstructured knowledge within it. Accordingly, we build HopRetriever which retrieves hops over Wikipedia to answer complex questions. Experiments on the HotpotQA dataset demonstrate that HopRetriever outperforms previously published evidence retrieval methods by large margins. Moreover, our approach also yields quantifiable interpretations of the evidence collection process., Accepted at AAAI 2021

The article examines the contradictions that caused the need for digital support for the process of learning the musical culture of China by future music teachers of Russia. In search of approaches to the development of such software, the authors come to the idea of the expediency of creating an electronic educational resource “Musical Culture of China”. The methodological basis for determining its content and structure was the musicological, psychological and pedagogical approaches. In their entirety, they provide a multi­vector characteristic of the identity of Chinese musical culture, its national characteristics. This resource has two levels, basic and advanced. When describing each of them, the justification of their block­ modular construction and the pedagogical purpose of all the selected structural components are given. At the basic level, students get acquainted with the predominantly knowledge component. The content of the advanced level is aimed at expanding the acquired knowledge and their practical application. The work with this electronic educational resource helps to increase the readiness of future teachers to implement the acquired knowledge and practical skills in their professional activities when introducing students to Chinese musical culture in musical­ instrumental, musical­plastic and vocal­choral activities.

BackgroundPrevious studies have suggested that pregnant women with pulmonary hypertension (PH) have high maternal mortality. However, indexes or factors that can predict maternal death are lacking.MethodsWe retrospectively reviewed pregnant women with PH admitted for delivery from 2012 to 2020 and followed them for over 6 months. The patients were divided into two groups according to 10-day survival status after delivery. Predictive models and predictors for maternal death were identified using four machine learning algorithms: naïve Bayes, random forest, gradient boosting decision tree (GBDT), and support vector machine.ResultsA total of 299 patients were included. The most frequent PH classifications were Group 1 PH (73.9%) and Group 2 PH (23.7%). The mortality within 10 days after delivery was 9.4% and higher in Group 1 PH than in the other PH groups (11.7 vs. 2.6%, P = 0.016). We identified 17 predictors, each with a P-value < 0.05 by univariable analysis, that were associated with an increased risk of death, and the most notable were pulmonary artery systolic pressure (PASP), platelet count, red cell distribution width, N-terminal brain natriuretic peptide (NT-proBNP), and albumin (all P < 0.01). Four prediction models were established using the candidate variables, and the GBDT model showed the best performance (F1-score = 66.7%, area under the curve = 0.93). Feature importance showed that the three most important predictors were NT-proBNP, PASP, and albumin.ConclusionMortality remained high, particularly in Group 1 PH. Our study shows that NT-proBNP, PASP, and albumin are the most important predictors of maternal death in the GBDT model. These findings may help clinicians provide better advice regarding fertility for women with PH.

Although lyrics generation has achieved significant progress in recent years, it has limited practical applications because the generated lyrics cannot be performed without composing compatible melodies. In this work, we bridge this practical gap by proposing a song rewriting system which rewrites the lyrics of an existing song such that the generated lyrics are compatible with the rhythm of the existing melody and thus singable. In particular, we propose SongRewriter,a controllable Chinese lyrics generation and editing system which assists users without prior knowledge of melody composition. The system is trained by a randomized multi-level masking strategy which produces a unified model for generating entirely new lyrics or editing a few fragments. To improve the controllabiliy of the generation process, we further incorporate a keyword prompt to control the lexical choices of the content and propose novel decoding constraints and a vowel modeling task to enable flexible end and internal rhyme schemes. While prior rhyming metrics are mainly for rap lyrics, we propose three novel rhyming evaluation metrics for song lyrics. Both automatic and human evaluations show that the proposed model performs better than the state-of-the-art models in both contents and rhyming quality., Comment: ACL Findings 2023

There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language. This work concerns addressing two major problems in existing Arabic PLMs which constraint progress of the Arabic NLU and NLG fields.First, existing Arabic PLMs are not well-explored and their pre-trainig can be improved significantly using a more methodical approach. Second, there is a lack of systematic and reproducible evaluation of these models in the literature. In this work, we revisit both the pre-training and evaluation of Arabic PLMs. In terms of pre-training, we explore improving Arabic LMs from three perspectives: quality of the pre-training data, size of the model, and incorporating character-level information. As a result, we release three new Arabic BERT-style models ( JABER, Char-JABER, and SABER), and two T5-style models (AT5S and AT5B). In terms of evaluation, we conduct a comprehensive empirical study to systematically evaluate the performance of existing state-of-the-art models on ALUE that is a leaderboard-powered benchmark for Arabic NLU tasks, and on a subset of the ARGEN benchmark for Arabic NLG tasks. We show that our models significantly outperform existing Arabic PLMs and achieve a new state-of-the-art performance on discriminative and generative Arabic NLU and NLG tasks. Our models and source code to reproduce of results will be made available shortly.

Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre-training process is computationally expensive due to the requirement of millions of video-text pairs and the redundant data structure of each video. To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training. To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also propose a non-parametric pooling mechanism to adaptively reweight the fine-grained video embedding conditioned on the text. Experimental results on text-video retrieval and video question answering show that the proposed LiteVL even outperforms previous video-language pre-trained models by a clear margin, though without any video-language pre-training., Comment: 13 pages, 6 figures, accepted by EMNLP 2022 main conference

Additional file 2: Figure S1. a protein sequence alignment of TgANT using MEGA7 from Toxoplasma gondii, Saccharomyces cerevisiae, Plasmodium falciparum, Cyclospora cayetanensis, Arabidopsis thaliana, Cardiosporidium cionae, Homo sapiens and other species. b Prediction results of TMHMM Server v.2.0 showed that TgANT has four transmembrane domains.

Recently, semantic parsing using hierarchical representations for dialog systems has captured substantial attention. Task-Oriented Parse (TOP), a tree representation with intents and slots as labels of nested tree nodes, has been proposed for parsing user utterances. Previous TOP parsing methods are limited on tackling unseen dynamic slot values (e.g., new songs and locations added), which is an urgent matter for real dialog systems. To mitigate this issue, we first propose a novel span-splitting representation for span-based parser that outperforms existing methods. Then we present a novel lexicon-injected semantic parser, which collects slot labels of tree representation as a lexicon, and injects lexical features to the span representation of parser. An additional slot disambiguation technique is involved to remove inappropriate span match occurrences from the lexicon. Our best parser produces a new state-of-the-art result (87.62%) on the TOP dataset, and demonstrates its adaptability to frequently updated slot lexicon entries in real task-oriented dialog, with no need of retraining.

Morality in dialogue systems has raised great attention in research recently. A moral dialogue system aligned with users' values could enhance conversation engagement and user connections. In this paper, we propose a framework, MoralDial to train and evaluate moral dialogue systems. In our framework, we first explore the communication mechanisms of morality and resolve expressed morality into three parts, which indicate the roadmap for building a moral dialogue system. Based on that, we design a simple yet effective method: constructing moral discussions between simulated specific users and the dialogue system. The constructed discussions consist of expressing, explaining, revising, and inferring moral views in dialogue exchanges, which makes conversational models learn morality well in a natural manner. Furthermore, we propose a novel evaluation method under the framework. We evaluate the multiple aspects of morality by judging the relation between dialogue responses and human values in discussions, where the multifaceted nature of morality is particularly considered. Automatic and manual experiments demonstrate that our framework is promising to train and evaluate moral dialogue systems., Comment: Accepted to ACL 2023

Supplemental material, sj-docx-1-onc-10.1177_11795549221116834 for Concordance of Genomic Profiles in Matched Tissue and Plasma Samples From Chinese Patients With Lung Cancer by Yueming He, Weifeng Guo, Meng Xu, Junling Huang, Xiange Zhang, Huanzhang Su, Dongxia Hong and Qun Liu in Clinical Medicine Insights: Oncology

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with $7\times$ fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks., Accepted to ACL 2022

Additional file 1: Table S1. Nucleotide sequences of primers used for RT-qPCR.

Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding~(VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that our Wukong-Reader has superior performance on various VDU tasks such as information extraction. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.

Recent work explored the potential of large-scale Transformer-based pre-trained models, especially Pre-trained Language Models (PLMs) in natural language processing. This raises many concerns from various perspectives, e.g., financial costs and carbon emissions. Compressing PLMs like BERT with negligible performance loss for faster inference and cheaper deployment has attracted much attention. In this work, we aim to explore larger compression ratios for PLMs, among which tensor decomposition is a potential but under-investigated one. Two decomposition and reconstruction protocols are further proposed to improve the effectiveness and efficiency during compression. Our compressed BERT with ${1}/{7}$ parameters in Transformer layers performs on-par with, sometimes slightly better than the original BERT in GLUE benchmark. A tiny version achieves $96.7\%$ performance of BERT-base with $ {1}/{48} $ encoder parameters (i.e., less than 2M parameters excluding the embedding layer) and $2.7 \times$ faster on inference. To show that the proposed method is orthogonal to existing compression methods like knowledge distillation, we also explore the benefit of the proposed method on a distilled BERT., Comment: Accepted at ICLR2022. Code available at https://github.com/twinkle0331/Xcompression

Supplemental material, sj-docx-1-onc-10.1177_11795549221116834 for Concordance of Genomic Profiles in Matched Tissue and Plasma Samples From Chinese Patients With Lung Cancer by Yueming He, Weifeng Guo, Meng Xu, Junling Huang, Xiange Zhang, Huanzhang Su, Dongxia Hong and Qun Liu in Clinical Medicine Insights: Oncology

In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models trained over a massive amount of dialogue data from scratch, we aim to build a powerful dialogue model with relatively fewer data and computation costs by inheriting valuable language capabilities and knowledge from PLMs. To this end, we train PanGu-Bot from the large PLM PANGU-alpha, which has been proven well-performed on a variety of Chinese natural language tasks. We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety. We show that PanGu-Bot outperforms state-of-the-art Chinese dialogue systems (CDIALGPT (Wang et al., 2020), EVA (Zhou et al., 2021), EVA2.0 (Gu et al., 2022)) w.r.t. the above three aspects. We also demonstrate that PanGu-Bot can be easily deployed to generate emotional responses without further training. Throughout our empirical analysis, we also point out that the PanGu-Bot response quality, knowledge correctness, and safety are still far from perfect, and further explorations are indispensable to building reliable and smart dialogue systems. Our model and code will be available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/PanGu-Bot soon., Comment: Update model and results; add comparison with EVA2.0

Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e.g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora. However, this Domain-Adaptive Pre-Training (DAPT; Gururangan et al. (2020)) tends to forget the previous general knowledge acquired by general PLMs, which leads to a catastrophic forgetting phenomenon and sub-optimal performance. To alleviate this problem, we propose a new framework of General Memory Augmented Pre-trained Language Model (G-MAP), which augments the domain-specific PLM by a memory representation built from the frozen general PLM without losing any general knowledge. Specifically, we propose a new memory-augmented layer, and based on it, different augmented strategies are explored to build the memory representation and then adaptively fuse it into the domain-specific PLM. We demonstrate the effectiveness of G-MAP on various domains (biomedical and computer science publications, news, and reviews) and different kinds (text classification, QA, NER) of tasks, and the extensive results show that the proposed G-MAP can achieve SOTA results on all tasks., Comment: EMNLP 2022,Long paper,Main conference

Recently, prompt tuning (PT) has gained increasing attention as a parameter-efficient way of tuning pre-trained language models (PLMs). Despite extensively reducing the number of tunable parameters and achieving satisfying performance, PT is training-inefficient due to its slow convergence. To improve PT's training efficiency, we first make some novel observations about the prompt transferability of "partial PLMs", which are defined by compressing a PLM in depth or width. We observe that the soft prompts learned by different partial PLMs of various sizes are similar in the parameter space, implying that these soft prompts could potentially be transferred among partial PLMs. Inspired by these observations, we propose Fast Prompt Tuning (FPT), which starts by conducting PT using a small-scale partial PLM, and then progressively expands its depth and width until the full-model size. After each expansion, we recycle the previously learned soft prompts as initialization for the enlarged partial PLM and then proceed PT. We demonstrate the feasibility of FPT on 5 tasks and show that FPT could save over 30% training computations while achieving comparable performance., Comment: Findings of EMNLP 2022

To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the HyperLink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multi-hop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios., Accepted by ACL 2022 main conference; The dataset and code are available at https://github.com/jzhoubu/HLP

The workflow of pretraining and fine-tuning has emerged as a popular paradigm for solving various NLP and V&L (Vision-and-Language) downstream tasks. With the capacity of pretrained models growing rapidly, how to perform parameter-efficient fine-tuning has become fairly important for quick transfer learning and deployment. In this paper, we design a novel unified parameter-efficient transfer learning framework that works effectively on both pure language and V&L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings as input, and outputs weights for fine-tuning different small modules in a pretrained language model, such as tuning the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning). We define a set of embeddings (e.g., layer, block, task and visual embeddings) as the key components to calculate hyper-embeddings, which thus can support both pure language and V&L tasks. Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V&L tasks confirm the effectiveness of our framework on both textual and visual modalities.

Additional file 1. Supplementary materials including methods and results.

The increasing size of generative Pre-trained Language Models (PLMs) has greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rates on GPT-2 and BART, respectively., ACL 2022

To proceed the clinical evaluation of DNA microarray for thalassemia gene detection.Peripheral blood samples of 166 thalassemia gene test subjects were collected and tested for thalassemia genes by microarray chip method and Gap-PCR method combined with PCR-reverse dot blot hybridization method according to double-blind control test. The specificity, sensitivity, positive predictive value, negative predictive value, and total coincidence rate of the microarray chip method were evaluated. When the two methods were inconsistent, multiplex ligation dependent probe amplification (MLPA) was used to verify the deletional α-thalassemia.Compared with Gap-PCR method, specificity, sensitivity, positive predictive value, negative predictive value, Youden index, and total coincidence rate of microarray chip method was 100% (70/70), 96.88% (93/96), 100% (93/93), 95.89% (70/73), 0.969, and 97.59% (162/166), respectively, while compared with PCR-reverse dot blot hybridization method was 100% (125/125), 100% (41/41), 100% (41/41), 100% (125/125), 1, and 100% (166/166), respectively.The microarray chip method for α-thalassemia gene detection shows the advantages of high specificity, sensitivity, and throughput.微阵列芯片法在α-地中海贫血检测中的方法学评价.对微阵列芯片法用于地中海贫血基因检测进行临床评价.收集166份地中海贫血基因受检者的外周血样本，按照双盲对照试验，分别应用微阵列芯片法和Gap-PCR法联合PCR-反向点杂交法对各样本进行地中海贫血基因检测，评价微阵列芯片法的特异性、灵敏度、阳性预测值、阴性预测值及总符合率。两种方法不一致时，对于缺失型α-地中海贫血采用多重连接依赖探针扩增（MLPA）法进行验证.微阵列芯片法与Gap-PCR法比较，特异性为100%（70/70），灵敏度为96.88%（93/96），阳性预测值为100%（93/93），阴性预测值为9589%（70/73），约登指数为0.969，总符合率为97.59%（162/166）；与PCR-反向点杂交法比较，其特异性为100%（125/125），灵敏度为100%（41/41），阳性预测值为100%（41/41），阴性预测值为100%（125/125），约登指数为1，总符合率为100%（166/166）.微阵列芯片法用于α-地中海贫血基因检测具有特异性高、灵敏度高、高通量等优点.

Although deep learning models are powerful among various applications, most deep learning models are still a black box, lacking verifiability and interpretability, which means the decision-making process that human beings cannot understand. Therefore, how to evaluate deep neural networks with explanations is still an urgent task. In this paper, we first propose a multi-semantic image recognition model, which enables human beings to understand the decision-making process of the neural network. Then, we presents a new evaluation index, which can quantitatively assess the model interpretability. We also comprehensively summarize the semantic information that affects the image classification results in the judgment process of neural networks. Finally, this paper also exhibits the relevant baseline performance with current state-of-the-art deep learning models.

To perform dried blood spots thalassemia gene detection in patients with positive blood phenotypes by microarray technology, and evaluate its value in clinical detection.DNA samples were extracted from dried blood spots of 410 patients. Microarray technology was used to detect 3 deletion and 3 non-deletion types of α-thalassemia and 19 β-thalassemia point mutations which were common gene mutions in China.There were 357 positive cases in all the 410 tested samples with the positive rate 87.07%, among which 299 cases (72.93%) carried deletion or point mutations of α-thalassemia, 29 cases (7.07%) carried point mutations of β-thalassemia and 29 cases (7.07%) carried gene mutations of complex αβ-thalassemia syndrome. The mutations of α-thalassemia were involved with --The most common genetic mutations are --微阵列芯片法在地中海贫血患者基因突变检测中应用的研究.应用微阵列芯片技术对血液学表型分析阳性的患者干血斑进行地中海贫血基因检测，评价其在临床检测中的价值。.对410例患者的干血斑样本进行DNA提取，采用微阵列芯片技术对中国人常见的3种缺失型及3种非缺失型α-地中海贫血、19种β-地中海贫血点突变进行基因检测。.410例样本中，共检出阳性357例，总阳性率为87.07%。单纯α-地中海贫血基因缺失和点突变的样本有299例，占72.93%（299/410），其中177例为广西柳州地区α、β地中海贫血基因主要突变类型分别为

Aiming at the problem of decreased accuracy in model classification resulting from the drift of data types in the few shot image classifications, this paper proposes a few shot image ensemble classification method based on meta-learning. Applying feature extraction network on the basis of attention mechanism to extract the features in the image, this method is able to input the feature vector extracted to meta-learner and generate the weight parameter of similarity measurement network in the corresponding channel herewith. And then integrate the similarity distance calculated by multiple weak classifiers in all channels to get the classification results of target sample. The experiment on the miniImageNet data set indicates that the tasks on the 5way1shot and 20way1shot have improved 11.8% and 13.41% respectively, corresponding to the control method, and that when applying the model parameters trained by MiniImageNet data set to perform the test on the CUB200 and Clatech101 data set, the classification accuracy of ECML proposed here is also higher than it of control model. Thus it can be seen that to generate a similarity measurement network by building a meta-learner on a meta-task not only improves the model's generalization ability, but also effectively solves the problem of performance degradation of few shot image classification when the data sample type drifts.

Magnetic CoFe2O4@carbon (CFO@C) nanoparticles were synthesized by employing glucose as carbon source via hydrothermal process, and their magnetic and electrochemical properties of CFO@C are both studied in this work. The Ms and Mr values of CFO@C nanoparticles are lower than those of pure CFO samples. The changed magnetic properties may be related to the carbon layer extinguishing the surface magnetic moment with spin canting. Benefiting from the amorphous structure and good electronic conductivity of carbon shells, the CFO@C 20 wt % electrode exhibited the capacity of 201 mA h g–1 at the current density of 500 mA g–1 and high reversible capacity up to 353 mA h g–1 after 100 cycles at the current density of 50 mA g–1, respectively.

The Ordos basin in North China may be one of the nonmarine basins with highest hydrocarbon generation per volume of sedimentary fill. It has great potential for unconventional oil and gas exploration, especially shale oil. When evaluating and predicting shale oil resources, the difference between different regions and different layers of shale is very obvious, so it cannot be calculated simply by average. A geostatistic decision tree was constructed based on a training set of 22 parameters (geochemistry, petrology, and sedimentology) for 110 shale samples from the basin, which can be used to implement genetic classification of subsequently collected shale samples. Three shale facies are identified by the decision tree. The properties of each facies are different, such as TOC, kerogen type and sulfur content. Genetic properties indicate formation conditions and provide base for the geostatistic assignments. The differential distribution of shale facies in the basin reflects their origins from different formation mechanism. Elements ratios, isotopes and biomarkers can be used to infer the sedimentary environment for each shale facies. Facies 1 mainly occurs in the upper Chang 7 section and shows evidence of quartz-rich source rock deposited under suboxic–anoxic condition. Facies 3 occurs at the lower Chang 7 section and shows evidence of clay-rich and pyrite-rich shale deposited under reducing dysoxic conditions with numerous higher plants input. Facies 1 and 2 constitutes the upper Chang 7 with interactive thick layers; Facies 2 and 3 constitute the lower Chang 7 with interactive thin layers. Petrology, geochemistry and sedimentology of the shale facies suggest: Facies 1 represents the suboxic–anoxic upper Chang 7 proximal quartz-rich shale, which has lower TOC and sulfur content than other facies. The organic lamina is straight but rare. Kerogen is type I and dominated by sapropel. The content of saturated hydrocarbon in chloroform asphalt “A” extract is the highest. Facies 2 has higher TOC value, sulfur content and soluble hydrocarbon content than facies 1. It was formed in unstable hydrodynamic environment under weak anoxic condition. It is characterized by abundant laminas with various deformation structures and more alginite and vitrinite in kerogen. Anoxic lower Chang 7 distal sulfur-rich shale source rock make up organic-rich facies 3 (∼7.0–17.0 wt % TOC). This facies received more higher-plant input than the other facies. Well preserved algae shells are common in numerous and flat laminas. The content of chloroform asphalt of facies 3 is higher than that of other facies, but among which the saturated hydrocarbon content is the lowest, the aromatic, non-hydrocarbon and asphalt are the highest, suggesting the fluidity of hydrocarbon is the worst. Compared to others, this facies was formed in the environment where temperature is relative lower, water is relative deeper, productivity and salinity are relative higher, and the reducibility is relative stronger. A deeper understanding of shale classification and distribution is important for the accurate resource evaluation and optimal exploration planning of shale oil.

Sinolimprichtia alpina var. dissecta is a plant variety which is characterized from S. alpina var. alpina in possessing characteristic, highly dissected bracteoles. In the current study, we have sequenced the complete chloroplast genome of S. alpina var. dissecta using the Illumina sequencing platform. The chloroplast genome is 156,719 bp in length, consisting of a LSC region of 95,625 bp, a SSC region of 10,500 bp, and a pair of inverted repeats (IR) regions of 25,297 bp. The GC content was 37.7%. A total of 126 unique genes were identified, including 81 protein-coding genes, 37 tRNA genes and 8 rRNA genes. Phylogenetic analysis based on 28 chloroplast genomes indicates that S. alpina var. dissecta is most closely related to Pterygopleurum neurophyllum.

Zhang, Hua-Jie, Zhang, Dai-Gui, Feng, Tao, Liu, Qun, Yang, Jing-Yuan, Chen, Li, Wang, Heng-Chang, Deng, Tao (2021): A new species of Potentilla (Potentilleae, Rosaceae) from central China, with reference to molecular and morphological evidence. Phytotaxa 480 (2): 185-194, DOI: 10.11646/phytotaxa.480.2.7, URL: http://dx.doi.org/10.11646/phytotaxa.480.2.7

Potentilla sunhangii D.G. Zhang, H.C. Wang & T. Deng, sp. nov. (Figs. 3, 4) Type: — CHINA. Hubei, Shennongjia Forest District alpine grasslands in Jinhou hill, E110°18′44.50″, N31°27′59.31″, elev. 2932 m, 13 June 2019, Q . Liu wlc05 (holotype KUN1498244; isotype KUN1498245). Diagnosis:— Potentilla sunhangii is similar to P. saundersiana, from which it is distinguished by many granular glands on both the abaxial and adaxial sides of leaves, sepals, epi-sepals and stipule, and leaf blade sparse appressed sericeous. Description: —Herbs perennial. Root terete, slender, with brown remaining stipules, membranous. Flowering stem erect, single or 2–3 cluster, 6–10 cm tall, with white long pubescence. Radical leaves palmately 3 foliolate, 1.5–4 cm including petiole. Petioles with white long pubescence, mid leaflet ovalellipsoidal, lateral leaflets obliquely ovoid, length and width are nearly equal, 0.6–1cm, apex obtuse, base cuneata or truncate, margin obtusely 10–12 serrate on side, green on both surfaces, both sides of leaflets with sparse pubescence and sticky glands. Cauline leaves 1–2, leaflets resembling those of radical leaves, smaller. Radical leaves stipule brown, membranous, with sparse pubescence. Cauline leaves, only radical concrescent with petiole, stipules grayish green, oval, obtuse, with sparse pubescence and granular glands. Inflorescence corymbose cyme, 1–3 flowered, pedicel 1.5–2 cm, densely with long pubescence and glands. Flowers diameter 1.5 cm-sepals long triangle, apex taper, episepals elliptic lanceolate, apex obtuse, abaxially sparsely pubescent and stick glands. Petals yellow, obcordate, apex micro concave, stamens ca. 20, filaments sparsely with pubescence, carpels 20, ovary sparsely pubescence, styles subterminal, base thickened, stigma bending. Distribution and habitat: — P. sunhangii is known only in Jin-hou-ling moutains, Shennongjia Fostery District, Hubei, China (Fig. 5). The terrain is mountainous with deep ravines and the climate is subtropical monsoon climate. The species is alpine herbs, which distributed in relatively drought alpine grassland meadow even in rock crevices. Etymology: — Potentilla sunhangii is named in honor of Prof. Hang Sun, a Chinese botanist who has made significant contributions to the flora of China and research on Biogeography., Published as part of Zhang, Hua-Jie, Zhang, Dai-Gui, Feng, Tao, Liu, Qun, Yang, Jing-Yuan, Chen, Li, Wang, Heng-Chang & Deng, Tao, 2021, A new species of Potentilla (Potentilleae, Rosaceae) from central China, with reference to molecular and morphological evidence, pp. 185-194 in Phytotaxa 480 (2) on pages 188-192, DOI: 10.11646/phytotaxa.480.2.7, http://zenodo.org/record/5416148

Trichosanthes sunhangii D.G.Zhang, Z.M.Li, Qun Liu & T.Deng, sp. nov. Type: CHINA, Hubei Province, Shennongjia Forestry District, 990 m, 28 July 2011, D. G . Zhang et al. 20110728049 (holotype KUN [barcode 1498246!]; isotypes KUN [barcode 1498247!], JIU!). Paratypes: CHINA, Hubei Province, Shennonjia Forestry District, 990 m, 8 Oct 2020, Qun Liu Qun gl001 (KUN), Qun gl002 (KUN). Lianas. Dioecious. Stems grooved, puberulent. Petiole 2.5���6.5 cm long, striate, puberulent; leaf blade adaxially pale green, glabrous; abaxially deep green, with long hairs at first, sparsely puberulent along veins at maturity, (6���)8���11(��� 14) �� (5���)7���11(���14) cm, papery, usually 3���5-lobed or unlobed; lobes oblong to triangular, margin denticulate, apex acuminate, veins sparsely puberulent. Tendrils 2 or 3, pubescent. Male raceme solitary, 3���5-flowered, or with solitary flowers in axillary pairs; peduncle 10���20 cm; robust, grooved, slightly hairy; bracts sessile, oblanceolate, 1.2���2.5 �� 0.5���1 cm, base attenuate, entire, sparsely puberulent, apex distally 3 parted, segments lanceolate; calyx tube narrowly cylindric, dilated at apex, 2.5���3(���3.5) cm, segments lanceolate, 5���8 �� 10��� 1.5 mm; corolla white, lobes obovate, 20 mm long, 16 mm wide, filiform tassels on both sides, pilose; anthers close together, about 5 mm long and 3 mm in diameter. Pistillate flowers solitary; pedicel 5���8 cm long, puberulent; calyx tube cylindric, 20���25 �� 5���8 mm; ovary oblong, 10���20 �� 5���10 mm, puberulent; corolla like male flower; ovary oval, green, 2 cm long, 1 cm diameter, style 2 cm long, stigmas 3, yellow. Fruiting peduncle 4���9 cm long; fruit orange-yellow, globose or oblong, 8���11 �� 8���11 cm, sparsely pubescent. Seeds, ovate-oblong, compressed, 1���1.5 �� 0.8���1 cm, marginate. Fl. Jun���Aug, fr. Aug���Oct. Figure 3. Trichosanthes sunhangii D.G.Zhang, Z.M.Li, Qun Liu & T.Deng. Left: Staminate plant. A, staminate flower dissection; B, habit; C, flower front view; D, bracts; E, bracts of T. kirilowii; F, bracts of T. rosthornii. Right: Pistillate plant: a, pistillate flower, leaf and tendril; b, fruit; c, seed (drawing by Jianing Yang). Figure 4. Maximum parsimony tree of Trichosanthes sunhangii and related species. The MP tree is based on the combined matrix of ITS, matK, rpl20-rps12 sequences. Numbers below branches are MP bootstraps and ML bootstraps, numbers above branches indicate Bayesian posterior probability; the new species is shown in bold. Distribution and habitat: Trichosanthes sunhangii is known only from near Qin Mountain, Yangri County, Xujiazhuang Songluo county and along the Longmen River; roadsides and thickets on limestone foothills at elevations of 900-1000 m. Vernacular name: ��t���������, sh��n n��ng ji�� gu�� l��u in Chinese Pinyin. Etymology: The species epithet honors Prof. Hang Sun (1963���), a Chinese Botanist who has conducted research on plant taxonomy, floristics, biogeography and evolutionary biology and who has inspired many people through his work., Published as part of Liu, Qun, Lin, Nan, Zhang, Dai-Gui, Huang, Xian-Han, Wang, Heng-Chang, Yang, Jing- Yuan, Tojibaev, Komiljon, Lv, Zhen-Yu, Deng, Tao & Li, Zhi-Min, 2021, Trichosanthes sunhangii (Cucurbitaceae), a new species from Hubei, China, pp. 287-294 in Phytotaxa 479 (3) on pages 290-292, DOI: 10.11646/phytotaxa.479.3.5, http://zenodo.org/record/5413934