Search

The fundamental understanding of sluggish hydrogen evolution reaction (HER) kinetics on a platinum (Pt) surface in alkaline media is a topic of considerable debate. Herein, we combine cyclic voltammetry (CV) and electrical transport spectroscopy (ETS) approaches to probe the Pt surface at different pH values and develop molecular-level insights into the pH-dependent HER kinetics in alkaline media. The change in HER Tafel slope from ∼110 mV/decade in pH 7-10 to ∼53 mV/decade in pH 11-13 suggests considerably enhanced kinetics at higher pH. The ETS studies reveal a similar pH-dependent switch in the ETS conductance signal at around pH 10, suggesting a notable change of surface adsorbates. Fixed-potential calculations and chemical bonding analysis suggest that this switch is attributed to a change in interfacial water orientation, shifting from primarily an O-down configuration below pH 10 to a H-down configuration above pH 10. This reorientation weakens the O-H bond in the interfacial water molecules and modifies the reaction pathway, leading to considerably accelerated HER kinetics at higher pH. Our integrated studies provide an unprecedented molecular-level understanding of the nontrivial pH-dependent HER kinetics in alkaline media.

Nonlinear optical processing of ambient natural light is highly desired for computational imaging and sensing. Strong optical nonlinear response under weak broadband incoherent light is essential for this purpose. By merging 2D transparent phototransistors (TPTs) with liquid crystal (LC) modulators, we create an optoelectronic neuron array that allows self-amplitude modulation of spatially incoherent light, achieving a large nonlinear contrast over a broad spectrum at orders-of-magnitude lower intensity than achievable in most optical nonlinear materials. We fabricated a 10,000-pixel array of optoelectronic neurons, and experimentally demonstrated an intelligent imaging system that instantly attenuates intense glares while retaining the weaker-intensity objects captured by a cellphone camera. This intelligent glare-reduction is important for various imaging applications, including autonomous driving, machine vision, and security cameras. The rapid nonlinear processing of incoherent broadband light might also find applications in optical computing, where nonlinear activation functions for ambient light conditions are highly sought.

The goal of No-Reference Image Quality Assessment (NR-IQA) is to predict the perceptual quality of an image in line with its subjective evaluation. To put the NR-IQA models into practice, it is essential to study their potential loopholes for model refinement. This paper makes the first attempt to explore the black-box adversarial attacks on NR-IQA models. Specifically, we first formulate the attack problem as maximizing the deviation between the estimated quality scores of original and perturbed images, while restricting the perturbed image distortions for visual quality preservation. Under such formulation, we then design a Bi-directional loss function to mislead the estimated quality scores of adversarial examples towards an opposite direction with maximum deviation. On this basis, we finally develop an efficient and effective black-box attack method against NR-IQA models. Extensive experiments reveal that all the evaluated NR-IQA models are vulnerable to the proposed attack method. And the generated perturbations are not transferable, enabling them to serve the investigation of specialities of disparate IQA models.

In this paper, we first investigate the global existence of a solution for the stochastic fractional nonlinear Schr\"odinger equation with radially symmetric initial data in a suitable energy space $H^{\alpha}$. We then show that the stochastic fractional nonlinear Schr\"odinger equation in the Stratonovich sense forms an infinite-dimensional stochastic Hamiltonian system, with its phase flow preserving symplecticity. Finally, we develop a stochastic midpoint scheme for the stochastic fractional nonlinear Schr\"odinger equation from the perspective of symplectic geometry. It is proved that the stochastic midpoint scheme satisfies the corresponding symplectic law in the discrete sense. A numerical example is conducted to validate the efficiency of the theory.

To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge. This challenge collects over 100 hours of multi-channel speech data recorded inside a new energy vehicle and 40 hours of noise for data augmentation. Two tracks, including automatic speech recognition (ASR) and automatic speech diarization and recognition (ASDR) are set up, using character error rate (CER) and concatenated minimum permutation character error rate (cpCER) as evaluation metrics, respectively. Overall, the ICMC-ASR Challenge attracts 98 participating teams and receives 53 valid results in both tracks. In the end, first-place team USTCiflytek achieves a CER of 13.16% in the ASR track and a cpCER of 21.48% in the ASDR track, showing an absolute improvement of 13.08% and 51.4% compared to our challenge baseline, respectively., Comment: Accepted at ICASSP 2024

Artificial intelligence (AI) technology has demonstrated remarkable potential in drug dis-covery, where pharmacokinetics plays a crucial role in determining the dosage, safety, and efficacy of new drugs. A major challenge for AI-driven drug discovery (AIDD) is the scarcity of high-quality data, which often requires extensive wet-lab work. A typical example of this is pharmacokinetic experiments. In this work, we develop a physical formula enhanced mul-ti-task learning (PEMAL) method that predicts four key parameters of pharmacokinetics simultaneously. By incorporating physical formulas into the multi-task framework, PEMAL facilitates effective knowledge sharing and target alignment among the pharmacokinetic parameters, thereby enhancing the accuracy of prediction. Our experiments reveal that PEMAL significantly lowers the data demand, compared to typical Graph Neural Networks. Moreover, we demonstrate that PEMAL enhances the robustness to noise, an advantage that conventional Neural Networks do not possess. Another advantage of PEMAL is its high flexibility, which can be potentially applied to other multi-task machine learning scenarios. Overall, our work illustrates the benefits and potential of using PEMAL in AIDD and other scenarios with data scarcity and noise.

Hydrazine-assisted water electrolysis offers a feasible path for low-voltage green hydrogen production. Herein, the design and synthesis of ultrathin RhRu0.5 -alloy wavy nanowires as bifunctional electrocatalysts for both the anodic hydrazine oxidation reaction (HzOR) and the cathodic hydrogen evolution reaction (HER) is reported. It is shown that the RhRu0.5 -alloy wavy nanowires can achieve complete electrooxidation of hydrazine with a low overpotential and high mass activity, as well as improved performance for the HER. The resulting RhRu0.5 bifunctional electrocatalysts enable, high performance hydrazine-assisted water electrolysis delivering a current density of 100 mA cm-2 at an ultralow cell voltage of 54 mV and a high current density of 853 mA cm-2 at a cell voltage of 0.6 V. The RhRu0.5  electrocatalysts further demonstrate a stable operation at a high current density of 100 mA cm-2 for 80 hours of testing period with little irreversible degradation. The overall performance greatly exceeds that of the previously reported hydrazine-assisted water electrolyzers, offering a pathway for efficiently converting hazardous hydrazine into molecular hydrogen.

Extensive macroscale two-dimensional (2-D) platinum (Pt) nanowire network (NWN) sheets are created through a hierarchical self-assembly process with the aid of biomolecular ligands. The Pt NWN sheet is assembled from the attachment growth of 1.9 nm-sized 0-D nanocrystals into 1-D nanowires featuring a high density of grain boundaries, which then interconnect to form monolayer network structures extending into centimeter-scale size. Further investigation into the formation mechanism reveals that the initial emergence of NWN sheets occurs at the gas/liquid interfaces of the bubbles produced by sodium borohydride (NaBH4) during the synthesis process. Upon the rupture of these bubbles, an exocytosis-like process releases the Pt NWN sheets at the gas/liquid surface, which subsequently merge into a continuous monolayer Pt NWN sheet. The Pt NWN sheets exhibit outstanding oxygen reduction reaction (ORR) activities, with specific and mass activities 12.0 times and 21.2 times greater, respectively, than those of current state-of-the-art commercial Pt/C electrocatalysts.

The accurate diagnosis of the compound fault of industrial robots can be highly beneficial to maintenance management. In the actual noisy working environment of industrial robots, the mixed and feeble failure features are easy to be overwhelmed, which poses a major challenge for the industrial robot compound fault diagnosis. Meanwhile, in the existing studies, a large-size deep learning model is the guarantee of decent denoising and fault diagnosis performance. However, this demands expensive computational costs and large data samples, which are not always available. In order to address both challenges, in this study, an integrated approach that contains two compact Transformer networks is proposed to achieve accurate compound fault diagnosis for industrial robots. In this approach, the feedback current signals collected from a six-axis industrial robot are first transformed into time-frequency image representation via continuous wavelet transformation (CWT). Secondly, a novel deep learning algorithm called compact Uformer is proposed to denoise the time-frequency image. Subsequently, the denoised time-frequency images are fed into compact convolutional Transformer (CCT) for compound fault diagnosis. An experimental study based on a real-world industrial robot compound fault dataset was conducted. The experimental results reveal that the proposed method can achieve satisfactory compound fault diagnosis accuracy based on the data collected from the noisy environment in comparison with the state-of-the-art algorithms.

The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge., Comment: 5 pages. Accepted at ICASSP2023

By incorporating additional contextual information, deep biasing methods have emerged as a promising solution for speech recognition of personalized words. However, for real-world voice assistants, always biasing on such personalized words with high prediction scores can significantly degrade the performance of recognizing common words. To address this issue, we propose an adaptive contextual biasing method based on Context-Aware Transformer Transducer (CATT) that utilizes the biased encoder and predictor embeddings to perform streaming prediction of contextual phrase occurrences. Such prediction is then used to dynamically switch the bias list on and off, enabling the model to adapt to both personalized and common scenarios. Experiments on Librispeech and internal voice assistant datasets show that our approach can achieve up to 6.7% and 20.7% relative reduction in WER and CER compared to the baseline respectively, mitigating up to 96.7% and 84.9% of the relative WER and CER increase for common cases. Furthermore, our approach has a minimal performance impact in personalized scenarios while maintaining a streaming inference pipeline with negligible RTF increase.

Contextual information plays a crucial role in speech recognition technologies and incorporating it into the end-to-end speech recognition models has drawn immense interest recently. However, previous deep bias methods lacked explicit supervision for bias tasks. In this study, we introduce a contextual phrase prediction network for an attention-based deep bias method. This network predicts context phrases in utterances using contextual embeddings and calculates bias loss to assist in the training of the contextualized model. Our method achieved a significant word error rate (WER) reduction across various end-to-end speech recognition models. Experiments on the LibriSpeech corpus show that our proposed model obtains a 12.1% relative WER improvement over the baseline model, and the WER of the context phrases decreases relatively by 40.5%. Moreover, by applying a context phrase filtering strategy, we also effectively eliminate the WER degradation when using a larger biasing list., Comment: Accepted by interspeech2023

While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the MLLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing MLLMs for the target MLLM. In this work, we for the first time investigate the VPG transferability across LLMs, and explore a solution to reduce the cost of VPG transfer. We first study the VPG transfer across different LLM sizes (e.g., small-to-large), and across different LLM types, through which we diagnose the key factors to maximize the transfer efficiency. Based on our observation, we design a two-stage transfer framework named VPGTrans, which is simple yet highly effective. Through extensive experiments, we demonstrate that VPGTrans helps significantly speed up the transfer learning process without compromising performance. Remarkably, it helps achieve the VPG transfer from BLIP-2 OPT$_\text{2.7B}$ to BLIP-2 OPT$_\text{6.7B}$ with over 10 times speed-up and 10.7% training data compared with connecting a VPG to OPT$_\text{6.7B}$ from scratch. Further, a series of intriguing findings and potential rationales behind them are provided and discussed. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs., Comment: Project Website: https://vpgtrans.github.io Code: https://github.com/VPGTrans/VPGTrans NeurIPS 2023

This study presents a novel approach to addressing the challenge of missing data in multivariate time series, with a particular focus on the complexities of healthcare data. Our Conditional Self-Attention Imputation (CSAI) model, grounded in a transformer-based framework, introduces a conditional hidden state initialization tailored to the intricacies of medical time series data. This methodology diverges from traditional imputation techniques by specifically targeting the imbalance in missing data distribution, a crucial aspect often overlooked in healthcare datasets. By integrating advanced knowledge embedding and a non-uniform masking strategy, CSAI adeptly adjusts to the distinct patterns of missing data in Electronic Health Records (EHRs).

Open-vocabulary keyword spotting (KWS), which allows users to customize keywords, has attracted increasingly more interest. However, existing methods based on acoustic models and post-processing train the acoustic model with ASR training criteria to model all phonemes, making the acoustic model under-optimized for the KWS task. To solve this problem, we propose a novel unified two-pass open-vocabulary KWS (U2-KWS) framework inspired by the two-pass ASR model U2. Specifically, we employ the CTC branch as the first stage model to detect potential keyword candidates and the decoder branch as the second stage model to validate candidates. In order to enhance any customized keywords, we redesign the U2 training procedure for U2-KWS and add keyword information by audio and text cross-attention into both branches. We perform experiments on our internal dataset and Aishell-1. The results show that U2-KWS can achieve a significant relative wake-up rate improvement of 41% compared to the traditional customized KWS systems when the false alarm rate is fixed to 0.5 times per hour., Comment: Accepted by ASRU2023

The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pix2seq). In this paper, we introduce a novel paradigm for object location modeling called pix2emb method, where we ask the LMM to output the location embeddings and then decode them with different decoders. This paradigm allows us to use different location formats (such as bounding boxes and masks) in multimodal conversations. Leveraging the proposed pix2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning. Comprehensive experiments show the effectiveness of our NExT-Chat on various tasks, e.g., NExT-Chat (87.7) vs. Shikra (86.9) on POPE-Random, NExT-Chat (68.9) vs. LISA (67.9) on referring expression segmentation task, and NExT-Chat (79.6) vs. Kosmos-2 (62.3) on region caption task. The code and model are released at https://github.com/NExT-ChatV/NExT-Chat., Comment: Technical Report (https://next-chatv.github.io/)

The attention-based deep contextual biasing method has been demonstrated to effectively improve the recognition performance of end-to-end automatic speech recognition (ASR) systems on given contextual phrases. However, unlike shallow fusion methods that directly bias the posterior of the ASR model, deep biasing methods implicitly integrate contextual information, making it challenging to control the degree of bias. In this study, we introduce a spike-triggered deep biasing method that simultaneously supports both explicit and implicit bias. Moreover, both bias approaches exhibit significant improvements and can be cascaded with shallow fusion methods for better results. Furthermore, we propose a context sampling enhancement strategy and improve the contextual phrase filtering algorithm. Experiments on the public WenetSpeech Mandarin biased-word dataset show a 32.0% relative CER reduction compared to the baseline model, with an impressively 68.6% relative CER reduction on contextual phrases., Comment: Accepted by ASRU2023

No-Reference Video Quality Assessment (NR-VQA) plays an essential role in improving the viewing experience of end-users. Driven by deep learning, recent NR-VQA models based on Convolutional Neural Networks (CNNs) and Transformers have achieved outstanding performance. To build a reliable and practical assessment system, it is of great necessity to evaluate their robustness. However, such issue has received little attention in the academic community. In this paper, we make the first attempt to evaluate the robustness of NR-VQA models against adversarial attacks, and propose a patch-based random search method for black-box attack. Specifically, considering both the attack effect on quality score and the visual quality of adversarial video, the attack problem is formulated as misleading the estimated quality score under the constraint of just-noticeable difference (JND). Built upon such formulation, a novel loss function called Score-Reversed Boundary Loss is designed to push the adversarial video's estimated quality score far away from its ground-truth score towards a specific boundary, and the JND constraint is modeled as a strict $L_2$ and $L_\infty$ norm restriction. By this means, both white-box and black-box attacks can be launched in an effective and imperceptible manner. The source code is available at https://github.com/GZHU-DVL/AttackVQA.

BACKGROUND: Genetic factors and smoking contribute to chronic obstructive pulmonary disease (COPD), but whether a combined polygenic risk score (PRS) is associated with incident COPD and whether it has a synergistic effect on smoking remains unclear. We aimed to investigate the association of the PRS with COPD and explore whether smoking behaviours could modify such association. METHODS: Multivariable Cox proportional hazards models were used to estimate hazard ratios (HRs) and 95% confidence intervals for the association of the PRS and smoking with COPD. RESULTS: The study included 439 255 participants (mean age 56.5 years; 53.9% female), with a median follow-up of 9.0 years. PRSlasso containing 2.5 million variants showed better discrimination and a stronger association for incident COPD than PRS279 containing 279 genome-wide significance variants. Compared with low genetic risk, the HRs of medium and high genetic risk were 1.39 (95% CI 1.31-1.48) and 2.40 (95% CI 2.24-2.56), respectively. The HR of high genetic risk and current smoking was 11.62 (95% CI 10.31-13.10) times that of low genetic risk and never smoking. There were significant interactions between PRSlasso and smoking status for incident COPD (pinteraction<0.001). From low genetic risk to high genetic risk, the HRs of current smoking increased from 4.32 (95% CI 3.69-5.06) to 6.89 (95% CI 6.21-7.64) and the population-attributable risks of smoking increased from 42.7% to 61.1%. CONCLUSIONS: The PRS constructed from millions of variants below genome-wide significance showed significant associations with incident COPD. Participants with a high genetic risk may be more susceptible to developing COPD when exposed to smoking. Copyright ©The authors 2022. For reproduction rights and permissions contact permissions@ersnet.org.

Background: Genetic variation increases the risk of lung cancer, but the extent to which smoking amplifies this effect remains unknown. Therefore, we aimed to investigate the risk of lung cancer in people with different genetic risks and smoking habits. Methods: This prospective cohort study included 345,794 European ancestry participants from the UK Biobank and followed up for 7.2 [6.5–7.8] years. Results: Overall, 26.2% of the participants were former smokers, and 9.8% were current smokers. During follow-up, 1687 (0.49%) participants developed lung cancer. High genetic risk and smoking were independently associated with an increased risk of incident lung cancer. Compared with never-smokers, HR per standard deviation of the PRS increase was 1.16 (95% CI, 1.11–1.22), and HR of heavy smokers (≥40 pack-years) was 17.89 (95% CI, 15.31–20.91). There were no significant interactions between the PRS and the smoking status or pack-years. Population-attributable fraction analysis showed that smoking cessation might prevent 76.4% of new lung cancers. Conclusions: Both high genetic risk and smoking were independently associated with higher lung cancer risk, but the increased risk of smoking was much more significant than heredity. The combination of traditional risk factors and additional PRS provides realistic application prospects for precise prevention. © 2022, The Author(s).

Background: Excessive production of mitochondrial reactive oxygen species (ROS) is a central mechanism for the development of diabetes complications. Recently, hypoxia has been identified to play an additional pathogenic role in diabetes. In this study, we hypothesized that ROS overproduction was secondary to the impaired responses to hypoxia due to the inhibition of hypoxia-inducible factor-1 (HIF-1) by hyperglycemia. Methods: The ROS levels were analyzed in the blood of healthy subjects and individuals with type 1 diabetes after exposure to hypoxia. The relation between HIF-1, glucose levels, ROS production and its functional consequences were analyzed in renal mIMCD-3 cells and in kidneys of mouse models of diabetes. Results: Exposure to hypoxia increased circulating ROS in subjects with diabetes, but not in subjects without diabetes. High glucose concentrations repressed HIF-1 both in hypoxic cells and in kidneys of animals with diabetes, through a HIF prolyl-hydroxylase (PHD)-dependent mechanism. The impaired HIF-1 signaling contributed to excess production of mitochondrial ROS through increased mitochondrial respiration that was mediated by Pyruvate dehydrogenase kinase 1 (PDK1). The restoration of HIF-1 function attenuated ROS overproduction despite persistent hyperglycemia, and conferred protection against apoptosis and renal injury in diabetes. Conclusions: We conclude that the repression of HIF-1 plays a central role in mitochondrial ROS overproduction in diabetes and is a potential therapeutic target for diabetic complications. These findings are timely since the first PHD inhibitor that can activate HIF-1 has been newly approved for clinical use. Funding: This work was supported by grants from the Swedish Research Council, Stockholm County Research Council, Stockholm Regional Research Foundation, Bert von Kantzows Foundation, Swedish Society of Medicine, Kung Gustaf V:s och Drottning Victorias Frimurarestifelse, Karolinska Institute's Research Foundation

Large-scale commonsense knowledge bases empower a broad range of AI applications, where the automatic extraction of commonsense knowledge (CKE) is a fundamental and challenging problem. CKE from text is known for suffering from the inherent sparsity and reporting bias of commonsense in text. Visual perception, on the other hand, contains rich commonsense knowledge about real-world entities, e.g., (person, can_hold, bottle), which can serve as promising sources for acquiring grounded commonsense knowledge. In this work, we present CLEVER, which formulates CKE as a distantly supervised multi-instance learning problem, where models learn to summarize commonsense relations from a bag of images about an entity pair without any human annotation on image instances. To address the problem, CLEVER leverages vision-language pre-training models for deep understanding of each image in the bag, and selects informative instances from the bag to summarize commonsense entity relations via a novel contrastive attention mechanism. Comprehensive experimental results in held-out and human evaluation show that CLEVER can extract commonsense knowledge in promising quality, outperforming pre-trained language model-based methods by 3.9 AUC and 6.4 mAUC points. The predicted commonsense scores show strong correlation with human judgment with a 0.78 Spearman coefficient. Moreover, the extracted commonsense can also be grounded into images with reasonable interpretability. The data and codes can be obtained at https://github.com/thunlp/CLEVER., Comment: Accepted by AAAI 2023

This paper summarizes the outcomes from the ISCSLP 2022 Intelligent Cockpit Speech Recognition Challenge (ICSRC). We first address the necessity of the challenge and then introduce the associated dataset collected from a new-energy vehicle (NEV) covering a variety of cockpit acoustic conditions and linguistic contents. We then describe the track arrangement and the baseline system. Specifically, we set up two tracks in terms of allowed model/system size to investigate resource-constrained and -unconstrained setups, targeting to vehicle embedded as well as cloud ASR systems respectively. Finally we summarize the challenge results and provide the major observations from the submitted systems., Comment: Accepted by ISCSLP2022

Video quality is a primary concern for video service providers. In recent years, the techniques of video quality assessment (VQA) based on deep convolutional neural networks (CNNs) have been developed rapidly. Although existing works attempt to introduce the knowledge of the human visual system (HVS) into VQA, there still exhibit limitations that prevent the full exploitation of HVS, including an incomplete model by few characteristics and insufficient connections among these characteristics. To overcome these limitations, this paper revisits HVS with five representative characteristics, and further reorganizes their connections. Based on the revisited HVS, a no-reference VQA framework called HVS-5M (NRVQA framework with five modules simulating HVS with five characteristics) is proposed. It works in a domain-fusion design paradigm with advanced network structures. On the side of the spatial domain, the visual saliency module applies SAMNet to obtain a saliency map. And then, the content-dependency and the edge masking modules respectively utilize ConvNeXt to extract the spatial features, which have been attentively weighted by the saliency map for the purpose of highlighting those regions that human beings may be interested in. On the other side of the temporal domain, to supplement the static spatial features, the motion perception module utilizes SlowFast to obtain the dynamic temporal features. Besides, the temporal hysteresis module applies TempHyst to simulate the memory mechanism of human beings, and comprehensively evaluates the quality score according to the fusion features from the spatial and temporal domains. Extensive experiments show that our HVS-5M outperforms the state-of-the-art VQA methods. Ablation studies are further conducted to verify the effectiveness of each module towards the proposed framework., Comment: 13 pages, 5 figures, Journal paper

Speech command recognition (SCR) has been commonly used on resource constrained devices to achieve hands-free user experience. However, in real applications, confusion among commands with similar pronunciations often happens due to the limited capacity of small models deployed on edge devices, which drastically affects the user experience. In this paper, inspired by the advances of discriminative training in speech recognition, we propose a novel minimize sequential confusion error (MSCE) training criterion particularly for SCR, aiming to alleviate the command confusion problem. Specifically, we aim to improve the ability of discriminating the target command from other commands on the basis of MCE discriminative criteria. We define the likelihood of different commands through connectionist temporal classification (CTC). During training, we propose several strategies to use prior knowledge creating a confusing sequence set for similar-sounding command instead of creating the whole non-target command set, which can better save the training resources and effectively reduce command confusion errors. Specifically, we design and compare three different strategies for confusing set construction. By using our proposed method, we can relatively reduce the False Reject Rate~(FRR) by 33.7% at 0.01 False Alarm Rate~(FAR) and confusion errors by 18.28% on our collected speech command set., Comment: Accepted by Interspeech 2022

Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL., Comment: Accepted by EMNLP 2022

Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks. However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs that are pre-trained to generate target tokens, such as BERT. It is still unknown whether and how discriminative PLMs, e.g., ELECTRA, can be effectively prompt-tuned. In this work, we present DPT, the first prompt tuning framework for discriminative PLMs, which reformulates NLP tasks into a discriminative language modeling problem. Comprehensive experiments on text classification and question answering show that, compared with vanilla fine-tuning, DPT achieves significantly higher performance, and also prevents the unstable problem in tuning large PLMs in both full-set and low-resource settings. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/DPT., Comment: Accepted by Findings of ACL 2022

Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a plug-and-play fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the enhanced dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The code and data are publicly available at https://github.com/waxnkw/IETrans-SGG.pytorch., Comment: ECCV 2022 (Oral)

© 2018 IEEE. In 2-D van der Waals heterostructures, interactions between atomic layers dramatically change the vibrational properties of the hybrid system and demonstrate several interesting phenomena that are absent in individual materials. In this paper, we have investigated the vibrational properties of the heterostructure between transition metal dichalcogenide (TMD) and hexagonal boron nitride (hBN) on gold film at low- and high-frequency ranges by Raman spectroscopy. Nineteen Raman modes have been observed from the sample, including a new interlayer coupling mode at 28.8 cm-1. Compared to reported experimental results of tungsten disulfide (WS2) on SiO2/Si substrates, the Raman spectrum for WS2 on hBN/Au emerges a blue shift of about 8 cm-1. Furthermore, a remarkable enhancement of Raman intensity can be obtained when tuning hBN thickness in the heterostructure. Through systematic first-principles calculations, numerical simulations, and analytical calculations, we find that the 28.8 cm-1 mode originates from the shearing motion between monolayer TMD and hBN layers. In addition, the gold substrate and hBN layers form an optical cavity and the cavity interference effects enhance the obtained Raman intensity. This paper demonstrates the novel vibrational modes of 2-D van der Waals heterostructure as an effective tool to characterize a variety of such heterostructures and reveals a new method to enhance the Raman response of 2-D materials.

The E26 transformation-specific or E-twenty-six (ETS) genes encode a superfamily of transcription factors involved in diverse biological processes. Here, we report the identification and characterization of a previously unidentified member of the ETS transcription factors, Spi2, that is found exclusively in the ray-finned fish kingdom. We show that the expression of spi2 is restricted to hemogenic endothelial cells (HECs) and to hematopoietic stem and progenitor cells (HSPCs) in zebrafish. Using bacteria artificial chromosome transgenesis, we generate a spi2 reporter line, TgBAC(spi2:P2a-GFP), which manifests the GFP pattern recapitulating the endogenous spi2 expression. Genetic ablation of spi2 has little effect on HEC formation and the endothelial-to-hematopoietic transition, but results in compromised proliferation of HSPCs in the caudal hematopoietic tissue (CHT) during early development and in severe myeloid lineage defect in adulthood. Epistatic analysis shows that spi2 acts downstream of runx1 in regulating HSPC development in the CHT. Our study identifies Spi2 as an essential regulator for definitive hematopoietic cell development and creates a TgBAC(spi2:P2a-GFP) reporter line for tracking HECs, HSPCs, myeloid cells and thrombocytes from early development to adulthood.

Background: To examine the association between regular use of proton pump inhibitors and the risk of incident dementia, including dementia subtypes, and whether the association differs between APOE genotypes. Methods: Based on a prospective analysis of data from the UK Biobank, 501,002 individuals (female, 54.4%) aged between 40 and 70 years, who had no prevalent dementia at baseline, were enrolled between 2006 and 2010 and followed up to 2018. We compared all-cause dementia, Alzheimer's disease (AD), and vascular dementia (VaD) incidence rates between proton pump inhibitor users and non-users by the Cox proportional hazard model. Results: During 4,438,839 person-years of follow-up (median length of follow-up, 9.0 years), there were 2505 incident cases of all-cause dementia, including 932 cases of AD and 524 cases of VaD. The incident rate of all-cause dementia among proton pump inhibitor users was 1.06 events per 1000 person-years, compared with 0.51 events per 1000 person-years among non-users. After adjustment for multiple confounders and indications, the hazard ratios (HRs) of the proton pump inhibitor users were 1.20 (95% CI, 1.07-1.35) for incident all-cause dementia, 1.23 (95% CI, 1.02-1.49) for incident AD, and 1.32 (95% CI, 1.05-1.67) for incident VaD. In addition, the association between proton pump inhibitor use and all-cause dementia differed by APOE genotype (P for interaction = 0.048). Among APOE epsilon 4 heterozygotes, the fully adjusted HR of proton pump inhibitor use was 1.46 (95% CI, 1.22-1.75) and 1.68 (95% CI, 1.36-2.07), especially for individuals aged 65 years and older. Conclusions: The finding of this large population-based cohort study indicates that the use of proton pump inhibitors is associated with an increased risk of incident dementia, particularly among APOE epsilon 4 heterozygotes.

© 2018 IEEE. In 2-D van der Waals heterostructures, interactions between atomic layers dramatically change the vibrational properties of the hybrid system and demonstrate several interesting phenomena that are absent in individual materials. In this paper, we have investigated the vibrational properties of the heterostructure between transition metal dichalcogenide (TMD) and hexagonal boron nitride (hBN) on gold film at low- and high-frequency ranges by Raman spectroscopy. Nineteen Raman modes have been observed from the sample, including a new interlayer coupling mode at 28.8 cm-1. Compared to reported experimental results of tungsten disulfide (WS2) on SiO2/Si substrates, the Raman spectrum for WS2 on hBN/Au emerges a blue shift of about 8 cm-1. Furthermore, a remarkable enhancement of Raman intensity can be obtained when tuning hBN thickness in the heterostructure. Through systematic first-principles calculations, numerical simulations, and analytical calculations, we find that the 28.8 cm-1 mode originates from the shearing motion between monolayer TMD and hBN layers. In addition, the gold substrate and hBN layers form an optical cavity and the cavity interference effects enhance the obtained Raman intensity. This paper demonstrates the novel vibrational modes of 2-D van der Waals heterostructure as an effective tool to characterize a variety of such heterostructures and reveals a new method to enhance the Raman response of 2-D materials.

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for large amounts of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/CPT., Comment: Work in progress

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

Object detection using automotive radars has not been explored with deep learning models in comparison to the camera based approaches. This can be attributed to the lack of public radar datasets. In this paper, we collect a novel radar dataset that contains radar data in the form of Range-Azimuth-Doppler tensors along with the bounding boxes on the tensor for dynamic road users, category labels, and 2D bounding boxes on the Cartesian Bird-Eye-View range map. To build the dataset, we propose an instance-wise auto-annotation method. Furthermore, a novel Range-Azimuth-Doppler based multi-class object detection deep learning model is proposed. The algorithm is a one-stage anchor-based detector that generates both 3D bounding boxes and 2D bounding boxes on Range-Azimuth-Doppler and Cartesian domains, respectively. Our proposed algorithm achieves 56.3% AP with IOU of 0.3 on 3D bounding box predictions, and 51.6% with IOU of 0.5 on 2D bounding box prediction. Our dataset and the code can be found at https://github.com/ZhangAoCanada/RADDet.git., Comment: Accepted by 18th Conference on Robots and Vision (CRV), CRV 2021

Scene graph generation aims to identify objects and their relations in images, providing structured image representations that can facilitate numerous applications in computer vision. However, scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. In this work, we propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. The intuition is that by aligning commonsense knowledge bases and images, we can automatically create large-scale labeled data to provide distant supervision for visual relation learning. To alleviate the noise in distantly labeled data, we further propose a framework that iteratively estimates the probabilistic relation labels and eliminates the noisy ones. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines. By further incorporating human-labeled data in a semi-supervised fashion, our model outperforms state-of-the-art fully supervised models by a large margin (e.g., 8.3 micro- and 7.8 macro-recall@50 improvements for predicate classification in Visual Genome evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/VisualDS., Comment: Accepted by ICCV 2021

Data augmentation has attracted a lot of research attention in the deep learning era for its ability in alleviating data sparseness. The lack of labeled data for unseen evaluation databases is exactly the major challenge for cross-domain text-to-SQL parsing. Previous works either require human intervention to guarantee the quality of generated data, or fail to handle complex SQL queries. This paper presents a simple yet effective data augmentation framework. First, given a database, we automatically produce a large number of SQL queries based on an abstract syntax tree grammar. For better distribution matching, we require that at least 80% of SQL patterns in the training data are covered by generated queries. Second, we propose a hierarchical SQL-to-question generation model to obtain high-quality natural language questions, which is the major contribution of this work. Finally, we design a simple sampling strategy that can greatly improve training efficiency given large amounts of generated data. Experiments on three cross-domain datasets, i.e., WikiSQL and Spider in English, and DuSQL in Chinese, show that our proposed data augmentation framework can consistently improve performance over strong baselines, and the hierarchical generation component is the key for the improvement.

Graph neural networks (GNNs) achieve remarkable performance for tasks on graph data. However, recent works show they are extremely vulnerable to adversarial structural perturbations, making their outcomes unreliable. In this paper, we propose DefenseVGAE, a novel framework leveraging variational graph autoencoders(VGAEs) to defend GNNs against such attacks. DefenseVGAE is trained to reconstruct graph structure. The reconstructed adjacency matrix can reduce the effects of adversarial perturbations and boost the performance of GCNs when facing adversarial attacks. Our experiments on a number of datasets show the effectiveness of the proposed method under various threat models. Under some settings it outperforms existing defense strategies. Our code has been made publicly available at https://github.com/zhangao520/defense-vgae., Comment: 11 pages, 2 figures

Atom-thin transition metal dichalcogenides (TMDs) have emerged as fascinating materials and key structures for electrocatalysis. So far, their edges, dopant heteroatoms and defects have been intensively explored as active sites for the hydrogen evolution reaction (HER) to split water. However, grain boundaries (GBs), a key type of defects in TMDs, have been overlooked due to their low density and large structural variations. Here, we demonstrate the synthesis of wafer-size atom-thin TMD films with an ultra-high-density of GBs, up to ~1012 cm−2. We propose a climb and drive 0D/2D interaction to explain the underlying growth mechanism. The electrocatalytic activity of the nanograin film is comprehensively examined by micro-electrochemical measurements, showing an excellent hydrogen-evolution performance (onset potential: −25 mV and Tafel slope: 54 mV dec−1), thus indicating an intrinsically high activation of the TMD GBs.

To reduce the emission of the carbon dioxide (CO2) in New Zealand (NZ), it is proposed to perform direct reduction (DR) of NZ titanomagnetite ironsand pellets using H2 gas. However, there is limited pre-existing knowledge on the DR behaviour of NZ titanomagnetite ironsand pellets in hydrogen. This thesis addresses this through an experimental investigation of the H2 gas-based reduction of Ar-sintered and pre-oxidised pellets, and an analytical kinetic model is developed to describe the reduction rate. The reduction rate of both types of pellets is found to increase with increasing temperature, H2 gas flow rate, and H2 gas concentration. For both pellets reduced at 1443 K above the critical flow rate, the maximum reduction degree reaches ~97% with a similar reduction rate. However, in the lower temperature range (< 1143 K), pre-oxidised pellets are reduced much faster than Ar-sintered ones. During reduction of both types of pellets, any TTH present is rapidly reduced first. After this step, TTM is then reduced to FeO, with Ti becoming enriched in the remaining unreduced TTM. FeO is further reduced to metallic Fe, which makes up to ~90% reduction degree. Eventually Ti-enrichment of the TTM leads to a change in the reduction pathway and it instead directly converts to metallic Fe and FeTiO3. Above ~90% reduction degree, reduction of the remaining Fe-Ti-O phases occurs (leading to the formation of TiO2 or (pseudobrookite) PSB/ferro-PSB). At the pellet-scale, both types of pellets present a single interface shrinking core phenomenon at higher temperatures. Metallic Fe is generated from pellet surface with a reaction interface moving inwards. However, at lower temperatures this pellet-scale interface becomes less defined in the pellets. Instead, grain-scale reaction fronts are observed.

This paper proposes a novel electric vehicle (EV) charging scheduling mechanism by controlling active and reactive charging power of EVs to provide joint voltage and frequency regulation (JVFR) in the distribution network. The objective of the scheduling problem is to minimize the net cost of the distribution system operator (DSO) for operating the distribution grid in the presence of both wind-based renewable generation and EV charging stations. We test the method on a 34-bus radial distribution network for different levels of VAR requirements of wind DG units to demonstrate the feasibility and performance of the proposed JVFR approach. The simulation results show that this JVFR greatly reduces the cost of DSO in comparison with Volt-VAR Optimization (VVO) technique, which is traditionally used to optimize voltage and minimize losses of a distribution network.

With the increasing load from the integration of electric vehicles (EVs) and the increasing uncertain supply from renewable energy sources (RES), it becomes increasingly challenging to maintain the reliability and stability of the power system, especially for its most vulnerable part, namely, the distribution network. Therefore, a larger amount of ancillary services are needed. Ancillary services are those services to balance the demand and supply so as to maintain the reliable operations of the power system. Leveraging the deferrable property and charging flexibilities of EVs, we propose to coordinate the charging power of EVs in the demand side to provide ancillary services. In doing so, the distribution system operator (DSO) can not only reduce energy costs and avoid overload of gird components, but also gain profits for providing ancillary services. In this thesis, we aim to minimize the net cost of the DSO by coordinating the charging of EVs to provide multiple ancillary services simultaneously in an optimal way. Specifically, in the first part of the thesis, we design a two-step strategy to provide frequency regulation and valley filling services jointly by coordinating the active charging power of EVs in distribution networks. Simulation results prove the feasibility and performance of the proposed strategy. In addition to controlling the active charging power of EVs, we can also utilize EVs to provide reactive power for voltage regulation, thanks to the recent advance of the EV charger technology. Thus in the second part of the thesis, we propose a joint voltage and frequency regulation strategy by coordinating both the active and reactive powers of EVs in distribution networks. The effectiveness and performance of the proposed strategy are demonstrated by numerical simulations as well.

Learning a classifier with control on the false-positive rate plays a critical role in many machine learning applications. Existing approaches either introduce prior knowledge dependent label cost or tune parameters based on traditional classifiers, which lack consistency in methodology because they do not strictly adhere to the false-positive rate constraint. In this paper, we propose a novel scoring-thresholding approach, tau-False Positive Learning (tau-FPL) to address this problem. We show the scoring problem which takes the false-positive rate tolerance into accounts can be efficiently solved in linear time, also an out-of-bootstrap thresholding method can transform the learned ranking function into a low false-positive classifier. Both theoretical analysis and experimental results show superior performance of the proposed tau-FPL over existing approaches., Comment: 32 pages, 3 figures. This is an extended version of our paper published in AAAI-18

In this study, controlled amount of dangling ends is introduced to the two series of poly(ethylene glycol)-based hydrogel networks with three and four crosslinking functionality by using click chemistry. The structure of the gels with regulated defect percentage is confirmed by comparing the results of low-field NMR characterization and Monte Carlo simulation. The mechanical properties of these gels were characterized by tensile stress–strain behaviors of the gels, and the results are analyzed by Gent model and Mooney–Rivlin model. The shear modulus of the swollen gels is found to be dependent on the functionality of the network, and decreases with the defect percentage. Furthermore, the value of shear modulus well obeys the Phantom model for all the gels with varied percentage of the defects. The maximum extension ratio, obtained from the fitting of Gent model, is also found to be dependent on the functionality of the network, and does not change with the defect percentage, except at very high defect percentage. The value of the maximum extension ratio is between that predicted from Phantom model and the Affine model. This indicates that at the large deformation, the fluctuation of the crosslinking points is suppressed for some extend but still exists. Polymer volume fractions at various defect percentages obtained from prediction of Flory–Rehner model are found to be in well agreement with the swelling experiment. All these results indicate that click chemistry is a powerful method to regulate the network structure and mechanical properties of the gels.

A series of physical double-network hydrogels is synthesized based on an amphiphilic triblock copolymer. The gel, which contains strong hydrophobic domains and sacrificial dynamic bonds of hydrogen bonds, is stiff and tough, and even stiffens in concentrated saline solution. Furthermore, due to its supramolecular structure, the gel features improved self-healing and self-recovery abilities.

Impaired expression of connexins, the gap junction subunits that facilitate direct cell-cell communication, have been implicated in prostate cancer growth. To elucidate the crucial role of connexins in prostate cancer progression, we performed a systematic quantitative RT-PCR screening of connexin expression in four representative prostate cancer cell lines across the spectrum of malignancy. Transcripts of several connexin subunits were detected in all four cell lines, and connexin 43 (Cx43) showed marked elevation at both RNA and protein levels in cells with increased metastatic potential. Analysis of gap-junction-mediated intercellular communication revealed homocellular coupling in PC-3 cells, which had the highest C x 43 expression, with minimal coupling in LNCaP cells where C x 43 expression was very low. Treatment with the gap junction inhibitor carbenoxolone or connexin mimetic peptide ACT-1 did not impair cell growth, suggesting that growth is independent of functional gap junctions. PC-3 cells with C x 43 expression reduced by shRNA showed decreased migration in monolayer wound healing assay, as well as decreased transwell invasion capacities when compared to control cells expressing non-targeting shRNA. These results, together with the correlation between C x 43 expression levels and the metastatic capacity of the cell lines, suggest a role of C x 43 in prostate cancer invasion and metastasis.

Impaired expression of connexins, the gap junction subunits that facilitate direct cell-cell communication, have been implicated in prostate cancer growth. To elucidate the crucial role of connexins in prostate cancer progression, we performed a systematic quantitative RT-PCR screening of connexin expression in four representative prostate cancer cell lines across the spectrum of malignancy. Transcripts of several connexin subunits were detected in all four cell lines, and connexin 43 (Cx43) showed marked elevation at both RNA and protein levels in cells with increased metastatic potential. Analysis of gap-junction-mediated intercellular communication revealed homocellular coupling in PC-3 cells, which had the highest C x 43 expression, with minimal coupling in LNCaP cells where C x 43 expression was very low. Treatment with the gap junction inhibitor carbenoxolone or connexin mimetic peptide ACT-1 did not impair cell growth, suggesting that growth is independent of functional gap junctions. PC-3 cells with C x 43 expression reduced by shRNA showed decreased migration in monolayer wound healing assay, as well as decreased transwell invasion capacities when compared to control cells expressing non-targeting shRNA. These results, together with the correlation between C x 43 expression levels and the metastatic capacity of the cell lines, suggest a role of C x 43 in prostate cancer invasion and metastasis.