Search

Organotypic three-dimensional liver spheroid cultures in which hepatic cells retain their molecular phenotype and functionality have emerged as powerful tools for preclinical drug development. In recent years a multitude of culture systems have been developed; however, a thorough side-by-side benchmarking of the different methods is lacking. Here, we compared the performance of ten different 96- and 384-well microplate types to support spheroid formation and long-term culture. Specifically, we evaluated differences in spheroid formation kinetics, viability, functionality, expression patterns, and their utility for hepatotoxicity assessments using primary human hepatocytes (PHH) and primary canine hepatocytes (PCH). All 96-well plates enabled formation of PHH liver spheroids, albeit with differences between plates in spheroid size, geometry, and reproducibility. Performance of different 384-wells was less consistent. Only 6/10 microplates supported the formation of PCH aggregates. Interestingly, even if PCH aggregates in these six microplates were more loosely packed than PHH spheroids, they maintained their function and were compatible with long-term pharmacological and toxicological assays. Overall, Corning and Biofloat plates showed the best performance in the formation of both human and canine liver spheroids with highest viability, most physiologically relevant phenotypes, superior CYP activity and lowest coefficient of variation in toxicity assays. The presented data constitutes a valuable resource that demonstrates the impacts of current ultra-low attachment plates on liver spheroid metrics and can guide evidence-based plate selection. Combined, these results have important implications for the cross-comparison of different studies and can facilitate the standardization and reproducibility of three-dimensional liver culture experiments. We assessed the performance of ten different 384- and 96-well ultra-low attachment (ULA) microplates in facilitating three-di

Understanding the early evolution of coronal mass ejections (CMEs), in particular their initiation, is the key to forecasting solar eruptions and induced disastrous space weather. Although many initiation mechanisms have been proposed, a full understanding of CME initiation, which is identified as a slow rise of CME progenitors in kinematics before the impulsive acceleration, remains elusive. Here, with a state-of-the-art thermal-magnetohydrodynamics simulation, we determine a complete CME initiation route in which multiple mainstream mechanisms occur in sequence yet are tightly coupled. The slow rise is first triggered and driven by the developing hyperbolic flux tube (HFT) reconnection. Subsequently, the slow rise continues as driven by the coupling of the HFT reconnection and the early development of torus instability. The end of the slow rise, i.e., the onset of the impulsive acceleration, is induced by the start of the fast magnetic reconnection coupled with the torus instability. These results unveil that the CME initiation is a complicated process involving multiple physical mechanisms, thus being hardly resolved by a single initiation mechanism., Comment: 35 pages, 15 figures, accepted for publication in ApJ

This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats, a crucial yet underexamined capability for their application as AI agents. Despite LLMs' advancements, existing benchmarks fail to assess their format-following proficiency adequately. FoFo fills this gap with a diverse range of real-world formats and instructions, developed through an AI-Human collaborative method. Our evaluation across both open-source (e.g., Llama 2, WizardLM) and closed-source (e.g., GPT-4, PALM2, Gemini) LLMs highlights three key findings: open-source models significantly lag behind closed-source ones in format adherence; LLMs' format-following performance is independent of their content generation quality; and LLMs' format proficiency varies across different domains. These insights suggest the need for specialized tuning for format-following skills and highlight FoFo's role in guiding the selection of domain-specific AI agents. FoFo is released here at https://github.com/SalesforceAIResearch/FoFo., Comment: The first two authors contributed equally

Yu-Bin Liu,1,2 Yun Liu,1,3 Lin Zhang,1 Xing-Chen Zhou,1 Bo-Yuan Ren,1 Can Zheng,1 Chun-Hua Hao,4 Wei-Ting Wang,4 Xia Xia,5 Gang-Qiao Zhou,1,3 Chu-Tse Wu,1 Ji-De Jin1 1Department of Experimental Hematology and Biochemistry, Beijing Institute of Radiation Medicine, Beijing, People’s Republic of China; 2Division of (Bio) Pharmaceutics, Institute of Zhejiang University - Quzhou, Zhejiang, People’s Republic of China; 3School of Medicine, Guizhou University, Guiyang, People’s Republic of China; 4Center for Pharmacodynamic Research, Tianjin Institute of Pharmaceutical Research, Tianjin, People’s Republic of China; 5Beijing SH Biotechnology Co., Ltd., Beijing, People’s Republic of ChinaCorrespondence: Chu-Tse Wu; Ji-de Jin, Email wuct@bmi.ac.cn; jinjide505@163.comBackground: Whether anticoagulant therapy should be used after spinal-cord injury (SCI) surgery was controversial. The anticoagulation characteristics of a newly developed anticoagulant, recombinant neorudin (EPR-hirudin (EH)), were explored using a rat model of SCI to provide a basis for clinical anticoagulation therapy of SCI.Methods: A rat model of SCI was developed by Allen’s method. Then, thrombosis in the inferior vena cava was induced by ligation. The low-bleeding characteristics of EH were explored by investigating dose–response and time–effect relationships, as well as multiple administration of EH, on thrombus formation complicated with SCI.Results: EH inhibited thrombosis in a dose-dependent manner by reducing the wet weight and dry weight of the thrombus. An inhibiting action of EH on thrombosis was most evident in the group given EH 2 h after SCI. After multiple intravenous doses of EH, thrombosis inhibition was improved to that observed with low molecular weight heparin (LMWH) (87% vs 90%). EH administration after SCI neither increased bleeding in the injured spine nor damaged to nerve function. Bleeding duration and activated partial thromboplastin time were increased in the high-dose

Technological innovation in the new energy vehicle industry is conducive to the achievement of China’s major strategic goal of ‘carbon peak and carbon neutrality’. This research involved an empirical study on the relevant data of 14 listed new energy vehicle companies from 2012 to 2019. It used the entropy weight method to obtain the technological innovation index through the four indicators of research and development (R&D) investment, fixed asset investment, intangible assets, and patent application volume. Taking fiscal subsidies and tax burdens as independent variables, a fixed effect model was used to analyze the impact of fiscal and taxation policies on technological innovation in the new energy vehicle industry. The research results show that financial subsidies will encourage new energy vehicle companies to carry out technological innovation, the tax burden has no significant impact on the technological innovation of new energy vehicle enterprises, the scale and age of enterprises, as well as the proportion of R&D personnel to the total number of employees, will all encourage new energy vehicle companies to carry out technological innovation. Based on this, we put forward specific suggestions on further improving the fiscal subsidy and tax incentive policies.

Xing-Chen Zhou,* Yu-Bo Huang,* Zhen Liu,* Hong-Jiao Wu, Hua-Zhi Huang, Yu Tian, Shuang-Wei Hong, Hui-Jie Hu, Li-Jiang Lv, Zhi-Zhen Lv The Third Clinical Medical College of Zhejiang Chinese Medical University, Hangzhou, Zhejiang, People’s Republic of China*These authors contributed equally to this workCorrespondence: Zhi-Zhen Lv; Li-Jiang Lv, The Third Clinical Medical College of Zhejiang Chinese Medical University, No. 548, Binwen Road, Hangzhou, 310053, People’s Republic of China, Tel +86 13750821128 ; +86 13958107858, Email lvzhizhen1992@163.com; lvlijiang0288@163.comBackground: Research on the brain mechanisms underlying manual therapy (MT)-induced analgesia has been conducted worldwide. However, no bibliometric analysis has been performed on functional magnetic resonance imaging (fMRI) studies of MT analgesia. To provide a theoretical foundation for the practical application of MT analgesia, this study examined the current incarnation, hotspots, and frontiers of fMRI-based MT analgesia research over the previous 20 years.Methods: All publications were obtained from the Science Citation Index-Expanded (SCI-E) of Web of Science Core Collection (WOSCC). We used CiteSpace 6.1.R3 to analyze publications, authors, cited authors, countries, institutions, cited journals, references, and keywords. We also evaluated keyword co-occurrences and timelines, and citation bursts. The search was conducted from 2002– 2022 and was completed within one day on October 7, 2022.Results: In total, 261 articles were retrieved. The total number of annual publications showed a fluctuating but overall increasing trend. Author B. Humphreys had the highest number of publications (eight articles) and J. E. Bialosky had the highest centrality (0.45). The United States of America (USA) was the country with the most publications (84 articles), accounting for 32.18% of all publications. Output institutions were mainly the University of Zurich, University of S

Existing instance segmentation models learn task-specific information using manual mask annotations from base (training) categories. These mask annotations require tremendous human effort, limiting the scalability to annotate novel (new) categories. To alleviate this problem, Open-Vocabulary (OV) methods leverage large-scale image-caption pairs and vision-language models to learn novel categories. In summary, an OV method learns task-specific information using strong supervision from base annotations and novel category information using weak supervision from image-captions pairs. This difference between strong and weak supervision leads to overfitting on base categories, resulting in poor generalization towards novel categories. In this work, we overcome this issue by learning both base and novel categories from pseudo-mask annotations generated by the vision-language model in a weakly supervised manner using our proposed Mask-free OVIS pipeline. Our method automatically generates pseudo-mask annotations by leveraging the localization ability of a pre-trained vision-language model for objects present in image-caption pairs. The generated pseudo-mask annotations are then used to supervise an instance segmentation model, freeing the entire pipeline from any labour-expensive instance-level annotations and overfitting. Our extensive experiments show that our method trained with just pseudo-masks significantly improves the mAP scores on the MS-COCO dataset and OpenImages dataset compared to the recent state-of-the-art methods trained with manual masks. Codes and models are provided in https://vibashan.github.io/ovis-web/., Comment: Accepted to CVPR 2023. Project site: https://vibashan.github.io/ovis-web

Text-to-image (T2I) models based on diffusion processes have achieved remarkable success in controllable image generation using user-provided captions. However, the tight coupling between the current text encoder and image decoder in T2I models makes it challenging to replace or upgrade. Such changes often require massive fine-tuning or even training from scratch with the prohibitive expense. To address this problem, we propose GlueGen, which applies a newly proposed GlueNet model to align features from single-modal or multi-modal encoders with the latent space of an existing T2I model. The approach introduces a new training objective that leverages parallel corpora to align the representation spaces of different encoders. Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation. By the alignment of various feature representations, the GlueNet allows for flexible and efficient integration of new functionality into existing T2I models and sheds light on X-to-image (X2I) generation., Comment: ICCV 2023

The properties of pre-eruptive structures and coronal mass ejections (CMEs) are characterized by those of their footpoints, the latter of which thus attract great interest. However, how to identify the footpoints of pre-eruptive structures and how to identify the footpoints with ground-based instruments, still remain elusive. In this work, we study an arc-shaped structure intruding in the sunspot umbra. It is located close to the (pre-)eruptive flux rope footpoint and is thus expected to help identify the footpoint. We analyse this arc-shaped structure, which we name as "sunspot scar", in a CME event on 2012 July 12 and in two CME events in observationally-inspired MHD simulations performed by OHM and MPI-AMRVAC. The sunspot scar has a more inclined magnetic field with a weaker vertical component and a stronger horizontal component relative to that in the surrounding umbra and manifests as a light bridge in the white light passband. The hot field lines anchored in the sunspot scar are spatially at the transition between the flux rope and the background coronal loops, and temporally in the process of the slipping reconnection which builds up the flux rope. The sunspot scar and its related light bridge mark the edge of the CME flux rope footpoint, and especially, the edge of the pre-eruptive flux rope footpoint in the framework of "pre-eruptive structures being flux ropes". Therefore, they provide a new perspective for the identification of pre-eruptive and CME flux rope footpoints, and also new methods for studying the properties and evolution of pre-eruptive structures and CMEs with photospheric observations only., Comment: 25 pages, 14 figures, accepted for publication in Astronomy & Astrophysics

We introduce Lemur and Lemur-Chat, openly accessible language models optimized for both natural language and coding capabilities to serve as the backbone of versatile language agents. The evolution from language chat models to functional language agents demands that models not only master human interaction, reasoning, and planning but also ensure grounding in the relevant environments. This calls for a harmonious blend of language and coding capabilities in the models. Lemur and Lemur-Chat are proposed to address this necessity, demonstrating balanced proficiencies in both domains, unlike existing open-source models that tend to specialize in either. Through meticulous pre-training using a code-intensive corpus and instruction fine-tuning on text and code data, our models achieve state-of-the-art averaged performance across diverse text and coding benchmarks among open-source models. Comprehensive experiments demonstrate Lemur's superiority over existing open-source models and its proficiency across various agent tasks involving human communication, tool usage, and interaction under fully- and partially- observable environments. The harmonization between natural and programming languages enables Lemur-Chat to significantly narrow the gap with proprietary models on agent abilities, providing key insights into developing advanced open-source agents adept at reasoning, planning, and operating seamlessly across environments. https://github.com/OpenLemur/Lemur

Large Language Models (LLMs) have become ubiquitous across various domains, transforming the way we interact with information and conduct research. However, most high-performing LLMs remain confined behind proprietary walls, hindering scientific progress. Most open-source LLMs, on the other hand, are limited in their ability to support longer sequence lengths, which is a key requirement for many tasks that require inference over an input context. To address this, we have trained XGen, a series of 7B parameter models on up to 8K sequence length for up to 1.5T tokens. We have also finetuned the XGen models on public-domain instructional data, creating their instruction-tuned counterparts (XGen-Inst). We open-source our models for both research advancements and commercial applications. Our evaluation on standard benchmarks shows that XGen models achieve comparable or better results when compared with state-of-the-art open-source LLMs. Our targeted evaluation on long sequence modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence open-source LLMs.

The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models are released at https://github.com/salesforce/ULIP., Comment: Accepted by CVPR 2023

Prompt tuning approaches, which learn task-specific soft prompts for a downstream task conditioning on frozen pre-trained models, have attracted growing interest due to its parameter efficiency. With large language models and sufficient training data, prompt tuning performs comparably to full-model tuning. However, with limited training samples in few-shot settings, prompt tuning fails to match the performance of full-model fine-tuning. In this work, we focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. Recognizing the good generalization capabilities of ensemble methods in low-data regime, we first experiment and show that a simple ensemble of model predictions based on different source prompts, outperforms existing multi-prompt knowledge transfer approaches such as source prompt fusion in the few-shot setting. Motivated by this observation, we further investigate model ensembles and propose Sample-specific Ensemble of Source Models (SESoM). SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs. Through this way, SESoM inherits the superior generalization of model ensemble approaches and simultaneously captures the sample-specific competence of each source prompt. We conduct experiments across a diverse set of eight NLP tasks using models of different scales (T5-{base, large, XL}) and find that SESoM consistently outperforms the existing models of the same as well as larger parametric scale by a large margin.

Yu-Bin Liu,1 Lin Zhang,1 Xing-Chen Zhou,1 Ying Zhou,1 Yun Liu,1 Can Zheng,1 Xiao Xu,1 Pan-Pan Geng,1 Chun-Hua Hao,2 Zhuan-You Zhao,2 Chu-Tse Wu,1 Ji-De Jin1 1Department of Experimental Hematology and Biochemistry, Beijing Institute of Radiation Medicine, Beijing, 100850, People’s Republic of China; 2Center for Pharmacodynamic Research, Tianjin Institute of Pharmaceutical Research, Tianjin, 300462, People’s Republic of ChinaCorrespondence: Chu-Tse Wu; Ji-De Jin, Department of Experimental Hematology and Biochemistry, Beijing Institute of Radiation Medicine, No. 27 Taiping Road, Beijing, People’s Republic of China, Tel +86 1086-68158312 ; +86 1086-66931425, Email wuct@bmi.ac.cn; jinjide505@163.comIntroduction: Recombinant neorudin (EPR-hirudin, EH) was developed through the addition of an EPR (Glu-Pro-Arg) peptide to the amino terminus of hirudin, which can be recognized and cut by coagulation factors XIa (FXIa) and/or Xa (FXa). In this study, the low-bleeding antithrombotic effects of EH were evaluated utilizing experimental models of thrombosis in rabbits and rats to provide a test basis for clinical trials.Methods: The bleeding risks of EH and hirudin were first compared in mice by the tail-clipping method, and then the antithrombotic activity of EH was investigated in a rabbit model of arteriovenous bypass thrombosis and a rat model of thrombotic cerebral infarction.Results: In mice, intravenous administration of EH at 1.5 mg/kg and 3 mg/kg did not affect the bleeding time compared with normal saline, while the administration of hirudin at 1.5 mg/kg prolonged the bleeding time by over 3 times the administration of normal saline. Furthermore, intravenous administration of EH had a significant dose-dependent inhibitory effect on the formation and development of arteriovenous bypass thrombosis and thrombotic cerebral infarction. Compared with an equimolar dose of hirudin, the antithrombotic effect of EH was similar, while the bleeding si

We propose value retrieval with arbitrary queries for form-like documents to reduce human effort of processing forms. Unlike previous methods that only address a fixed set of field items, our method predicts target value for an arbitrary query based on the understanding of the layout and semantics of a form. To further boost model performance, we propose a simple document language modeling (SimpleDLM) strategy to improve document understanding on large-scale model pre-training. Experimental results show that our method outperforms previous designs significantly and the SimpleDLM further improves our performance on value retrieval by around 17% F1 score compared with the state-of-the-art pre-training method. Code is available at https://github.com/salesforce/QVR-SimpleDLM.

Despite great progress in object detection, most existing methods work only on a limited set of object categories, due to the tremendous human effort needed for bounding-box annotations of training data. To alleviate the problem, recent open vocabulary and zero-shot detection methods attempt to detect novel object categories beyond those seen during training. They achieve this goal by training on a pre-defined base categories to induce generalization to novel objects. However, their potential is still constrained by the small set of base categories available for training. To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs. Our method leverages the localization ability of pre-trained vision-language models to generate pseudo bounding-box labels and then directly uses them for training object detectors. Experimental results show that our method outperforms the state-of-the-art open vocabulary detector by 8% AP on COCO novel categories, by 6.3% AP on PASCAL VOC, by 2.3% AP on Objects365 and by 2.8% AP on LVIS. Code is available at https://github.com/salesforce/PB-OVD., Comment: ECCV 2022

Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training on the model with such data. However, given the limited size and concentrated focus of the gender-neutral data, catastrophic forgetting would occur during second-phase pre-training. Forgetting information in the original training data may damage the model's downstream performance by a large margin. In this work, we empirically show that catastrophic forgetting occurs in such methods by evaluating them with general NLP tasks in GLUE. Then, we propose a new method, GEnder Equality Prompt (GEEP), to improve gender fairness of pre-trained models with less forgetting. GEEP freezes the pre-trained model and learns gender-related prompts with gender-neutral data. Empirical results show that GEEP not only achieves SOTA performances on gender fairness tasks, but also forgets less and performs better on GLUE by a large margin., Comment: This paper has been accepted at the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)

China’s time-honored brands have a rich cultural heritage, carrying the history and culture of China, and they have been recognized and supported by the majority of consumers, forming strong reputations and influence. In the era of rapid development of the Internet and the impact of the new epidemic situation on the global economy, China’s time-honored brands not only usher in rare development opportunities but also face severe difficulties. This paper takes Quanjude, a well-known Chinese time-honored brand that has existed for more than 100 years, as the research object. Quanjude is loved by heads of state, government officials, people from all walks of life, and tourists at home and abroad, and it is referred to as “the first food in China.” This study used PEST, the five forces model, SWOT, and other strategic analysis tools to analyze the corporate culture and development strategy of Quanjude in detail. The results revealed it is not difficult to find that the advantages of Quanjude outweigh the disadvantages. With the expansion of the same industry, the brand characteristics of Quanjude gradually fade, and the homogenization trend begins to appear; consumers can more easily find substitutes in the industry, which will occupy a part of the market share. Quanjude’s brand, with its high popularity, and the Chinese government’s strong support for the development of time-honored enterprises can overcome their disadvantages and give full play to their advantages.

Lahore and Faisalabad are two highly populated metropolitans of Pakistan. The increasing number of vehicles and urbanization resulted in the contamination of urban road dust with heavy metals, which can expose residents to health hazards. The aim of this research was to determine the road dust heavy metal (Cu, Ni, Cd, Pb, and Zn) concentrations, their distribution, comparison, and their potential health risk assessment for local people. Herein, two sampling methods, namely deposit gauge method (DGM) and sweeping tools method (STM), were used to collect road dust samples in eight similar major road intersections. Dust collected by DGM (gravitational pull of air dust) showed higher heavy metals concentration than STM in both cities, which might be associated to the smaller dust particles collected in the DGM. Among sites, concentrations of heavy metals were the highest in dust obtained from the General Bus Stand in both cities. Dominant metals were Zn and Pb with an average concentration of 95.5 and 56.8 mg kg−1, respectively while Cd was the lowest in Lahore. Similarly, the concentration of Pb and Zn were also high in Faisalabad with an average of 90.4 and 49.5 mg kg−1 and Cd was the lowest. Overall, Lahore showed higher pollution levels than Faisalabad. The risk assessment results of the DGM indicated that there were potential carcinogenic health risks associated with Cd, as his values were close or exceeded the permissible limit of 1 × 10−4. Non-carcinogenic risk assessment results were lower than value 1 for both cities which means there were no non-carcinogenic hazards for the residents. Higher Enrichment ratio (ER) and geo accumulation index (Igeo) showed increasing heavy metal pollution in both cities is a serious concern. Therefore this research provides a vital base and help to authorities for pollution control strategies, remediation, and regular monitoring in order to improve eco-sustainable cities and health of urbanites. © 2019 Elsevier Ltd

How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization. It is well-known that in language data corpus, words follow a heavy-tail distribution. A large proportion of words appear only very few times and the embeddings of rare words are usually poorly optimized. We argue that such embeddings carry inadequate semantic signals, which could make the data utilization inefficient and slow down the pre-training of the entire model. To mitigate this problem, we propose Taking Notes on the Fly (TNF), which takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. Specifically, TNF maintains a note dictionary and saves a rare word's contextual information in it as notes when the rare word occurs in a sentence. When the same rare word occurs again during training, the note information saved beforehand can be employed to enhance the semantics of the current sentence. By doing so, TNF provides better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words in the sentences. We implement TNF on both BERT and ELECTRA to check its efficiency and effectiveness. Experimental results show that TNF's training time is $60\%$ less than its backbone pre-training models when reaching the same performance. When trained with the same number of iterations, TNF outperforms its backbone methods on most of downstream tasks and the average GLUE score. Source code is attached in the supplementary material., Comment: Qiyu Wu and Chen Xing contribute equally

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.

Fitting complex patterns in the training data, such as reasoning and commonsense, is a key challenge for language pre-training. According to recent studies and our empirical observations, one possible reason is that some easy-to-fit patterns in the training data, such as frequently co-occurring word combinations, dominate and harm pre-training, making it hard for the model to fit more complex information. We argue that mis-predictions can help locate such dominating patterns that harm language understanding. When a mis-prediction occurs, there should be frequently co-occurring patterns with the mis-predicted word fitted by the model that lead to the mis-prediction. If we can add regularization to train the model to rely less on such dominating patterns when a mis-prediction occurs and focus more on the rest more subtle patterns, more information can be efficiently fitted at pre-training. Following this motivation, we propose a new language pre-training method, Mis-Predictions as Harm Alerts (MPA). In MPA, when a mis-prediction occurs during pre-training, we use its co-occurrence information to guide several heads of the self-attention modules. Some self-attention heads in the Transformer modules are optimized to assign lower attention weights to the words in the input sentence that frequently co-occur with the mis-prediction while assigning higher weights to the other words. By doing so, the Transformer model is trained to rely less on the dominating frequently co-occurring patterns with mis-predictions while focus more on the rest more complex information when mis-predictions occur. Our experiments show that MPA expedites the pre-training of BERT and ELECTRA and improves their performances on downstream tasks., Comment: Accepted at SSL workshop at NeurIPS 2021

Deep neural networks (DNNs) are poorly calibrated when trained in conventional ways. To improve confidence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE). DBLE bases its confidence estimation on distances in the representation space. In DBLE, we first adapt prototypical learning to train classification models. It yields a representation space where the distance between a test sample and its ground truth class center can calibrate the model's classification performance. At inference, however, these distances are not available due to the lack of ground truth labels. To circumvent this by inferring the distance for every test sample, we propose to train a confidence model jointly with the classification model. We integrate this into training by merely learning from mis-classified training samples, which we show to be highly beneficial for effective learning. On multiple datasets and DNN architectures, we demonstrate that DBLE outperforms alternative single-model confidence calibration approaches. DBLE also achieves comparable performance with computationally-expensive ensemble approaches with lower computational cost and lower number of parameters.

Metric-based meta-learning techniques have successfully been applied to few-shot classification problems. In this paper, we propose to leverage cross-modal information to enhance metric-based few-shot learning methods. Visual and semantic feature spaces have different structures by definition. For certain concepts, visual features might be richer and more discriminative than text ones. While for others, the inverse might be true. Moreover, when the support from visual information is limited in image classification, semantic representations (learned from unsupervised text corpora) can provide strong prior knowledge and context to help learning. Based on these two intuitions, we propose a mechanism that can adaptively combine information from both modalities according to new image categories to be learned. Through a series of experiments, we show that by this adaptive combination of the two modalities, our model outperforms current uni-modality few-shot learning methods and modality-alignment methods by a large margin on all benchmarks and few-shot scenarios tested. Experiments also show that our model can effectively adjust its focus on the two modalities. The improvement in performance is particularly large when the number of shots is very small.

In this Letter, we investigate the long-duration quasi-static evolution of 12 pre-eruptive filaments (4 active region and 8 quiescent filaments), mainly focusing on the evolution of the filament height in three dimension (3D) and the decay index of the background magnetic field. The filament height in 3D is derived through two-perspective observations of \textit{Solar Dynamics Observatory} and \textit{Solar TErrestrial RElations Observatory}. The coronal magnetic field is reconstructed using the potential field source surface model. A new finding is that the filaments we studied show two types of long-duration evolution: one type is comprised of a long-duration static phase and a short slow rise phase with a duration of less than 12 hours and a speed of 0.1--0.7 km s$^{-1}$, while the other one only presents a slow rise phase but with an extremely long duration of more than 60 hours and a smaller speed of 0.01--0.2 km s$^{-1}$. At the moment approaching the eruption, the decay index of the background magnetic field at the filament height is similar for both active region and quiescent filaments. The average value and upper limit are $\sim$0.9 and $\sim$1.4, close to the critical index of torus instability. Moreover, the filament height and background magnetic field strength are also found to be linearly and exponentially related with the filament length, respectively., Comment: 6 pages, 4 figures, 1 table, accepted for publication in ApJ Letters

This paper reviews the first challenge on efficient perceptual image enhancement with the focus on deploying deep learning models on smartphones. The challenge consisted of two tracks. In the first one, participants were solving the classical image super-resolution problem with a bicubic downscaling factor of 4. The second track was aimed at real-world photo enhancement, and the goal was to map low-quality photos from the iPhone 3GS device to the same photos captured with a DSLR camera. The target metric used in this challenge combined the runtime, PSNR scores and solutions' perceptual results measured in the user study. To ensure the efficiency of the submitted models, we additionally measured their runtime and memory requirements on Android smartphones. The proposed solutions significantly improved baseline results defining the state-of-the-art for image enhancement on smartphones.

We present novel empirical observations regarding how stochastic gradient descent (SGD) navigates the loss landscape of over-parametrized deep neural networks (DNNs). These observations expose the qualitatively different roles of learning rate and batch-size in DNN optimization and generalization. Specifically we study the DNN loss surface along the trajectory of SGD by interpolating the loss surface between parameters from consecutive \textit{iterations} and tracking various metrics during training. We find that the loss interpolation between parameters before and after each training iteration's update is roughly convex with a minimum (\textit{valley floor}) in between for most of the training. Based on this and other metrics, we deduce that for most of the training update steps, SGD moves in valley like regions of the loss surface by jumping from one valley wall to another at a height above the valley floor. This 'bouncing between walls at a height' mechanism helps SGD traverse larger distance for small batch sizes and large learning rates which we find play qualitatively different roles in the dynamics. While a large learning rate maintains a large height from the valley floor, a small batch size injects noise facilitating exploration. We find this mechanism is crucial for generalization because the valley floor has barriers and this exploration above the valley floor allows SGD to quickly travel far away from the initialization point (without being affected by barriers) and find flatter regions, corresponding to better generalization., Comment: First two authors contributed equally

Yan-Min Long,1,2 Li-Gang Hu,1,3 Xue-Ting Yan,1,3 Xing-Chen Zhao,1,3 Qun-Fang Zhou,1,3 Yong Cai,2,4 Gui-Bin Jiang1,3 1State Key Laboratory of Environmental Chemistry and Ecotoxicology, Research Center for Eco-Environmental Sciences, Beijing, China; 2Institute of Environment and Health, Jianghan University, Wuhan, Hubei, China; 3College of Resources and Environment, University of Chinese Academy of Sciences, Beijing, China; 4Department of Chemistry and Biochemistry, Southeast Environmental Research Center, Florida International University, Miami, FL, USA Abstract: Understanding the mechanism of nanosilver-dependent antibacterial activity against microorganisms helps optimize the design and usage of the related nanomaterials. In this study, we prepared four kinds of 10 nm-sized silver nanoparticles (AgNPs) with dictated surface chemistry by capping different ligands, including citrate, mercaptopropionic acid, mercaptohexanoic acid, and mercaptopropionic sulfonic acid. Their surface-dependent chemistry and antibacterial activities were investigated. Owing to the weak bond to surface Ag, short carbon chain, and low silver ion attraction, citrate-coated AgNPs caused the highest silver ion release and the strongest antibacterial activity against Escherichia coli, when compared to the other tested AgNPs. The study on the underlying antibacterial mechanisms indicated that cellular membrane uptake of Ag, NAD+/NADH ratio increase, and intracellular reactive oxygen species (ROS) generation were significantly induced in both AgNP and silver ion exposure groups. The released silver ions from AgNPs inside cells through a Trojan-horse-type mechanism were suggested to interact with respiratory chain proteins on the membrane, interrupt intracellular O2 reduction, and induce ROS production. The further oxidative damages of lipid peroxidation and membrane breakdown caused the lethal effect on E. coli. Altogether, this study demonstrated that AgNPs exerted antibacterial activit

We study multi-turn response generation in chatbots where a response is generated according to a conversation context. Existing work has modeled the hierarchy of the context, but does not pay enough attention to the fact that words and utterances in the context are differentially important. As a result, they may lose important information in context and generate irrelevant responses. We propose a hierarchical recurrent attention network (HRAN) to model both aspects in a unified framework. In HRAN, a hierarchical attention mechanism attends to important parts within and among utterances with word level attention and utterance level attention respectively. With the word level attention, hidden vectors of a word level encoder are synthesized as utterance vectors and fed to an utterance level encoder to construct hidden representations of the context. The hidden vectors of the context are then processed by the utterance level attention and formed as context vectors for decoding the response. Empirical studies on both automatic evaluation and human judgment show that HRAN can significantly outperform state-of-the-art models for multi-turn response generation.

We study the problem of response selection for multi-turn conversation in retrieval-based chatbots. The task requires matching a response candidate with a conversation context, whose challenges include how to recognize important parts of the context, and how to model the relationships among utterances in the context. Existing matching methods may lose important information in contexts as we can interpret them with a unified framework in which contexts are transformed to fixed-length vectors without any interaction with responses before matching. The analysis motivates us to propose a new matching framework that can sufficiently carry the important information in contexts to matching and model the relationships among utterances at the same time. The new framework, which we call a sequential matching framework (SMF), lets each utterance in a context interacts with a response candidate at the first step and transforms the pair to a matching vector. The matching vectors are then accumulated following the order of the utterances in the context with a recurrent neural network (RNN) which models the relationships among the utterances. The context-response matching is finally calculated with the hidden states of the RNN. Under SMF, we propose a sequential convolutional network and sequential attention network and conduct experiments on two public data sets to test their performance. Experimental results show that both models can significantly outperform the state-of-the-art matching methods. We also show that the models are interpretable with visualizations that provide us insights on how they capture and leverage the important information in contexts for matching., Comment: Submitted to Computational Linguistics

We study response selection for multi-turn conversation in retrieval-based chatbots. Existing work either concatenates utterances in context or matches a response with a highly abstract context vector finally, which may lose relationships among utterances or important contextual information. We propose a sequential matching network (SMN) to address both problems. SMN first matches a response with each utterance in the context on multiple levels of granularity, and distills important matching information from each pair as a vector with convolution and pooling operations. The vectors are then accumulated in a chronological order through a recurrent neural network (RNN) which models relationships among utterances. The final matching score is calculated with the hidden states of the RNN. An empirical study on two public data sets shows that SMN can significantly outperform state-of-the-art methods for response selection in multi-turn conversation., Comment: ACL 2017

While automatic response generation for building chatbot systems has drawn a lot of attention recently, there is limited understanding on when we need to consider the linguistic context of an input text in the generation process. The task is challenging, as messages in a conversational environment are short and informal, and evidence that can indicate a message is context dependent is scarce. After a study of social conversation data crawled from the web, we observed that some characteristics estimated from the responses of messages are discriminative for identifying context dependent messages. With the characteristics as weak supervision, we propose using a Long Short Term Memory (LSTM) network to learn a classifier. Our method carries out text representation and classifier learning in a unified framework. Experimental results show that the proposed method can significantly outperform baseline methods on accuracy of classification., Comment: Accepted paper at COLING 2016

We consider incorporating topic information into the sequence-to-sequence framework to generate informative and interesting responses for chatbots. To this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The model utilizes topics to simulate prior knowledge of human that guides them to form informative and interesting responses in conversation, and leverages the topic information in generation by a joint attention mechanism and a biased generation probability. The joint attention mechanism summarizes the hidden vectors of an input message as context vectors by message attention, synthesizes topic vectors by topic attention from the topic words of the message obtained from a pre-trained LDA model, and let these vectors jointly affect the generation of words in decoding. To increase the possibility of topic words appearing in responses, the model modifies the generation probability of topic words by adding an extra probability item to bias the overall distribution. Empirical study on both automatic evaluation metrics and human annotations shows that TA-Seq2Seq can generate more informative and interesting responses, and significantly outperform the-state-of-the-art response generation models.

Nan Sha,1,2,* Hao Xu,1,2,* Tao Chen,1,2,* Da-wei Tian,1,2 Wan-qin Xie,3 Lin-Guo Xie,1,2 Yu Zhang,1,2 Chen Xing,1,2 Xiao-teng Liu,1,2 Zhong-Hua Shen,1,2 Zhou-Liang Wu,1,2 Hai-Long Hu1,2 Chang-Li Wu1,2 1Department of Urology, The Second Hospital of Tianjin Medical University, 2Tianjin Key Laboratory of Urology, Tianjin Institute of Urology, Tianjin, 3Key Laboratory of Genetics and Birth Health of Hunan province, Family Planning Research Institute of Hunan Province, Changsha, Hunan, People’s Republic of China *These authors contributed equally to this work Objective: The objective of this article was to summarize the relationship between some components of metabolic syndrome (MetS) and the histopathologic findings in bladder cancer in a Chinese population. Methods: We retrospectively analyzed data of 323 patients from the Department of Urology, Second Hospital of Tianjin Medical University between January 2012 and January 2014. All the patients were diagnosed with bladder cancer for the first time. Age, height, weight, histologic stage, grade, the presence of hypertension, diabetes mellitus, and body mass index were evaluated. The 2009 American Joint Committee on Cancer TNM staging system was used, with Ta and T1 tumors accepted as lower stage and T2, T3, and T4 tumors as higher stage bladder cancers. Also, pathologists assigned tumor grade according to the 1973 World Health Organization grading system. Noninvasive papillary urothelial neoplasms of low malignant potential were regarded as low grade. Analyses were completed using chi-square tests and logistic regression analysis. Results: Of the 323 patients, 164 had hypertension, 151 had diabetes mellitus, and 213 had a body mass index ≥25 kg/m2. MetS was significantly associated with histologic grade (P<0.001) and stage (P=0.006) of bladder cancer. Adjusted for age in binary logistic regression analysis, the presence of MetS predicts the risk of higher T stage (odds ratio =4.029, P&amp

Yu Zhang,1,* Linguo Xie,1,* Tao Chen,1,* Wanqin Xie,2 Zhouliang Wu,1 Hao Xu,1 Chen Xing,1 Nan Sha,1 Zhonghua Shen,1 Yunkai Qie,1 Xiaoteng Liu,1 Hailong Hu,1 Changli Wu1 1Department of Urology, The Second Hospital of Tianjin Medical University, Tianjin Institute of Urology, Tianjin, 2Key Laboratory of Genetics and Birth Health of Hunan Province, The Family Planning Research Institute of Hunan Province, Changsha, People’s Republic of China *These authors contributed equally to this work Objective: The management of stage 1 and grade 3 (T1G3) bladder cancer continues to be controversial. Although the transurethral resection of bladder tumor (TURBT) followed by intravesical chemotherapy is a conservative strategy for treatment of T1G3 bladder cancer, a relatively high risk of tumor recurrence and progression remains regarding the therapy. This study aimed to compare the efficacy of intravenous chemotherapy combined with intravesical chemotherapy versus intravesical chemotherapy alone for T1G3 bladder cancer after TURBT surgery. Methods: We retrospectively reviewed the cases of 457 patients who were newly diagnosed with T1G3 bladder urothelial carcinoma between January 2009 and March 2014. After TURBT, 281 patients received intravesical chemotherapy alone, whereas 176 patients underwent intravesical chemotherapy in combination with intravenous chemotherapy. Tumor recurrence and progression were monitored periodically by urine cytology and cystoscopy in follow-up. Recurrence-free survival and progression-free survival of the two chemotherapy strategies following TURBT were analyzed. Univariable and multivariable Cox hazards analyses were performed to predict the prognostic factors for tumor recurrence and progression. Results: The tumor recurrence rate was 36.7% for patients who received intravesical chemotherapy alone after TURBT, compared with 19.9% for patients who received intravenous chemotherapy combined with intravesical chemotherapy after TURBT (P&lt

Liquid scintillator (LS) will be adopted as the detector material in JUNO (Jiangmen Underground Neutrino Observatory). The energy resolution requirement of JUNO is 3%, which has never previously been reached. To achieve this energy resolution, the light yield of liquid scintillator is an important factor. PPO (the fluor) and bis-MSB (the wavelength shifter) are the two main materials dissolved in LAB. To study the influence of these two materials on the transmission of scintillation photons in LS, 25 and 12 cm-long quartz vessels were used in a light yield experiment. LS samples with different concentration of PPO and bis-MSB were tested. At these lengths, the light yield growth is not obvious when the concentration of PPO is higher than 4 g/L. The influence from bis-MSB becomes insignificant when its concentration is higher than 8 mg/L. This result could provide some useful suggestions for the JUNO LS., Comment: 5 pages, 9 figures, submitted to CPC

Nan Sha,* Linguo Xie,* Tao Chen,* Chen Xing, Xiaoteng Liu, Yu Zhang, Zhonghua Shen, Hao Xu, Zhouliang Wu, Hailong Hu, Changli Wu Department of Urology, Tianjin Key Laboratory of Urology, Tianjin Institute of Urology, Second Hospital of Tianjin Medical University, Tianjin, People’s Republic of China *These authors contributed equally to this work Objective: To evaluate the clinical significance of lymphovascular invasion (LVI) on recurrence and progression rates in patients with pT1 urothelial carcinoma of bladder after transurethral resection.Methods: This retrospective study was performed with 155 patients with newly diagnosed pT1 urothelial carcinoma of bladder who were treated with transurethral resection of bladder tumor at our institution from January 2006 to January 2010. The presence or absence of LVI was examined by pathologists. Chi-square test was performed to identify the correlations between LVI and other clinical and pathological features. Kaplan–Meier method was used to estimate the recurrence-free survival (RFS) and progression-free survival curves and difference was determined by the log-rank test. Univariate and multivariate analyses were performed to determine the predictive factors through a Cox proportional hazards analysis model.Results: LVI was detected in a total of 34 patients (21.9%). While LVI was associated with high-grade tumors (P<0.001) and intravesical therapy (P=0.009). Correlations with age (P=0.227), sex (P=0.376), tumor size (P=0.969), tumor multiplicity (P=0.196), carcinoma in situ (P=0.321), and smoking (P=0.438) were not statistically significant. There was a statistically significant tendency toward higher recurrence rate and shorter RFS time in LVI-positive patients. However, no statistically significant differences were observed in progression rate between the two groups. Moreover, multivariate Cox proportional hazards analysis revealed that LVI, tumor size, and smoking were independent pro

Aims Alternative transcription and splicing of the allograft inflammatory factor-1 (AIF-1) gene results in the expression of two different proteins: AIF-1 and interferon responsive transcript-1 (IRT-1). Here we explore the impact of AIF-1 and IRT-1 on vascular smooth muscle cell (VSMC) activation and neointima formation, the mechanisms underlying their alternative splicing, and associations of AIF-1 and IRT-1 mRNA with parameters defining human atherosclerotic plaque phenotype.Methods and results Translation of AIF-1 and IRT-1 results in different products with contrasting cellular distribution and functions. Overexpression of AIF-1 stimulates migration and proliferation of human VSMCs, whereas IRT-1 exerts opposite effects. Adenoviral infection of angioplasty-injured rat carotid arteries with AdAIF-1 exacerbates intima hyperplasia, whereas infection with AdIRT-1 reduces neointima. Expression of these variants is modulated by changes in nuclear factor of activated T-cells (NFAT) activity. Pharmacological inhibition of NFAT or targeting of NFATc3 with siRNA lowers the AIF-1/IRT-1 ratio and favors an anti-proliferative outcome. NFAT acts as a repressor on the IRT-1 transcriptional start site, which is also sensitive to interferon-γ stimulation. Expression of AIF-1 mRNA in human carotid plaques associates with less extracellular matrix and a more pro-inflammatory plaque and plasma profile, features that may predispose to plaque rupture. In contrast, expression of IRT-1 mRNA associates with a less aggressive phenotype and less VSMCs at the most stenotic region of the plaque.Conclusions Inhibition of NFAT signaling, by shifting the AIF-1/IRT-1 ratio, may be an attractive target to regulate the VSMC response to injury and manipulate plaque stability in atherosclerosis.

This thesis presents the development of a novel hybrid pneumatic-electric actuator which combines the advantages of both pneumatic and electrical actuator. The hybrid actuator consists of a pneumatic cylinder and a DC motor. They are connected in parallel using gears. The components are sized to provide the torque required to rotate a single-link robot arm vertically upwards. On/off solenoid valves are used rather than servo valves to keep the hardware cost low. A mathematical model of the nonlinear actuator dynamics is derived using a combination of physical laws and empirical curve fitting. The dynamics of the mechanical, electrical and pneumatic elements are included. Then a novel discrete-valued model-predictive control plus integral compensator algorithm is created for controlling the position of the pneumatic cylinder using the on/off valves. The control algorithm for the hybrid actuator is completed by using a conventional PD algorithm to control the electric motor. Experiments shows the hybrid actuator outperform pneumatic actuator in every aspect. Conversely, the DC motor added a faster acting and finer quantized force to the pneumatic cylinder force, which greatly improved the dynamic position control performance of the hybrid actuator. In experiments, the mean root-mean-square error and the maximum absolute error improved by 84% and 77%, respectively., Master of Science in Mechanical Engineering (MSME)

Human metallothionein-3 (hMT3), also named as human neuronal growth inhibitory factor (hGIF), can inhibit the outgrowth of embryonic cortical neurons in the presence of brain extracts. In order to systematically study the structure-property-reactivity-function relationship of hGIF, our laboratory designed a series of mutants and studied their structure, property, reactivity and functions by a series of chemical and biological tools including UV spectroscopy, CD spectroscopy, NMR, chemical reaction and primary neuronal culture assays. In summary, we concluded that the bioactivity of hGIF was regulated by multiple factors, including the (CPCP9)-C-6 motif, an additional threonine insert at sequence position 5, domain-domain interactions, the structure and stability of the metal-thiolate cluster and the linker. Our studies provide more and more evidences which revealed that the bioactivity of hGIF is mainly related to the essential metal release and its characteristic conformation. (C) 2008 Elsevier Inc. All rights reserved.

During this program Science Research Laboratory (SRL) and the Plasma Processing Group in the Department of Chemical Engineering at MIT are developing a large-area, directed plasma/atomic beam source for diamond deposition. The plasma source is based on an inductively-driven plasma accelerator that efficiently produces a high density (l0(exp 14-10(exp 17)/cu cm) plasma over an area of 0.1-1 square meters. The goal of this effort is to experimentally demonstrate the technical feasibility of employing the plasma source for high-throughput diamond deposition, through characterization of plasma parameters and preliminary diamond deposition experiments. A reactor design study will also be completed during Phase I, leading to an engineering design of a large-area plasma reactor for Phase II implementation. The period of performance is from 30 September 1994 to 29 May 1995. jg p.2, Original contains color plates: All DTIC and NTIS reproductions will be in black and white.