Search

Since the onset of the coronavirus disease 2019 (COVID-19) pandemic, SARS-CoV-2 variants capable of breakthrough infections have attracted global attention. These variants have significant mutations in the receptor-binding domain (RBD) of the spike protein and the membrane (M) protein, which may imply an enhanced ability to evade immune responses. In this study, an examination of co-mutations within the spike RBD and their potential correlation with mutations in the M protein was conducted. The EVmutation method was utilized to analyze the distribution of the mutations to elucidate the relationship between the mutations in the spike RBD and the alterations in the M protein. Additionally, the Sequence-to-Sequence Transformer Model (S2STM) was employed to establish mapping between the amino acid sequences of the spike RBD and M proteins, offering a novel and efficient approach for streamlined sequence analysis and the exploration of their interrelationship. Certain mutations in the spike RBD, G339D-S373P-S375F and Q493R-Q498R-Y505, are associated with a heightened propensity for inducing mutations at specific sites within the M protein, especially sites 3 and 19/63. These results shed light on the concept of mutational synergy between the spike RBD and M proteins, illuminating a potential mechanism that could be driving the evolution of SARS-CoV-2.

Yang J, Li K, Chen J, Hu X, Wang H, Zhu X. Onco Targets Ther. 2020;13:1979–1991. We, the Editor and Publisher of the journal OncoTargets and Therapy are retracting the published article. Following publication of the article, the authors raised concerns about the duplication of images from Figure 5. Specifically, The images for Figure 5G, HepG2 and Huh 7, LINC00460+miR-342-3p have been duplicated. The corresponding author was cooperative and responded to our queries and explained how the images came to be duplicated but was unable to provide satisfactory original data for the study. As verifying the validity of published work is core to the integrity of the scholarly record, the Publisher and Editor requested to retract the article and the authors agreed with this decision. We have been informed in our decision-making by our editorial policies and COPE guidelines. The retracted article will remain online to maintain the scholarly record, but it will be digitally watermarked on each page as “Retracted”.

The fund investment industry heavily relies on the expertise of fund managers, who bear the responsibility of managing portfolios on behalf of clients. With their investment knowledge and professional skills, fund managers gain a competitive advantage over the average investor in the market. Consequently, investors prefer entrusting their investments to fund managers rather than directly investing in funds. For these investors, the primary concern is selecting a suitable fund manager. While previous studies have employed quantitative or qualitative methods to analyze various aspects of fund managers, such as performance metrics, personal characteristics, and performance persistence, they often face challenges when dealing with a large candidate space. Moreover, distinguishing whether a fund manager's performance stems from skill or luck poses a challenge, making it difficult to align with investors' preferences in the selection process. To address these challenges, this study characterizes the requirements of investors in selecting suitable fund managers and proposes an interactive visual analytics system called FMLens . This system streamlines the fund manager selection process, allowing investors to efficiently assess and deconstruct fund managers' investment styles and abilities across multiple dimensions. Additionally, the system empowers investors to scrutinize and compare fund managers' performances. The effectiveness of the approach is demonstrated through two case studies and a qualitative user study. Feedback from domain experts indicates that the system excels in analyzing fund managers from diverse perspectives, enhancing the efficiency of fund manager evaluation and selection. IEEE

Unhealthy ageing poses a global challenge with profound healthcare and socioeconomic implications. Slowing down the ageing process offers a promising approach to reduce the burden of a number of age-related diseases, such as dementia, promoting healthy longevity in the old population. In response to the challenge of the ageing population and with a view to the future, Norway and the UK are fostering collaborations, supported by a "Money Follows Cooperation agreement" between the two nations. The inaugural Norway-UK joint meeting on ageing and dementia gathered leading experts on ageing and dementia from the two nations to share their latest discoveries in related fields. Since ageing is an international challenge, and to foster collaborations, we also invited leading scholars from 11 additional countries to join this event. This report provides a summary of the conference, highlighting recent progress on molecular ageing mechanisms, genetic risk factors, DNA damage and repair, mitophagy, autophagy, as well as progress on a series of clinical trials (e.g., using NAD+ precursors). The meeting facilitated dialogue among policy makers, administrative leaders, researchers, and clinical experts, aiming to promote international research collaborations and to translate findings into clinical applications and interventions to advance healthy ageing.

Sapce-borne gravitational wave antennas, such as LISA and LISA-like mission (Taiji and Tianqin), will offer novel perspectives for exploring our Universe while introduce new challenges, especially in data analysis. Aside from the known challenges like high parameter space dimension, superposition of large number of signals and etc., gravitational wave detections in space would be more seriously affected by anomalies or non-stationarities in the science measurements. Considering the three types of foreseeable non-stationarities including data gaps, transients (glitches), and time-varying noise auto-correlations, which may come from routine maintenance or unexpected disturbances during science operations, we developed a deep learning model for accurate signal extractions confronted with such anomalous scenarios. Our model exhibits the same performance as the current state-of-the-art models do for the ideal and anomaly free scenario, while shows remarkable adaptability in extractions of coalescing massive black hole binary signal against all three types of non-stationarities and even their mixtures. This also provide new explorations into the robustness studies of deep learning models for data processing in space-borne gravitational wave missions., Comment: 11 pages, 10 figures, 5 tables

Unhealthy aging poses a global challenge with profound healthcare and socioeconomic implications. Slowing down the aging process offers a promising approach to reduce the burden of a number of age-related diseases, such as dementia, and promoting healthy longevity in the old population. In response to the challenge of the aging population and with a view to the future, Norway and the United Kingdom are fostering collaborations, supported by a “Money Follows Cooperation agreement” between the 2 nations. The inaugural Norway–UK joint meeting on aging and dementia gathered leading experts on aging and dementia from the 2 nations to share their latest discoveries in related fields. Since aging is an international challenge, and to foster collaborations, we also invited leading scholars from 11 additional countries to join this event. This report provides a summary of the conference, highlighting recent progress on molecular aging mechanisms, genetic risk factors, DNA damage and repair, mitophagy, autophagy, as well as progress on a series of clinical trials (eg, using NAD+ precursors). The meeting facilitated dialogue among policymakers, administrative leaders, researchers, and clinical experts, aiming to promote international research collaborations and to translate findings into clinical applications and interventions to advance healthy aging., Unhealthy aging poses a global challenge with profound healthcare and socioeconomic implications. Slowing down the aging process offers a promising approach to reduce the burden of a number of age-related diseases, such as dementia, and promoting healthy longevity in the old population. In response to the challenge of the aging population and with a view to the future, Norway and the United Kingdom are fostering collaborations, supported by a "Money Follows Cooperation agreement" between the 2 nations. The inaugural Norway-UK joint meeting on aging and dementia gathered leading experts on aging and dementia from the 2 nations to share their latest discoveries in related fields. Since aging is an international challenge, and to foster collaborations, we also invited leading scholars from 11 additional countries to join this event. This report provides a summary of the conference, highlighting recent progress on molecular aging mechanisms, genetic risk factors, DNA damage and repair, mitophagy, autophagy, as well as progress on a series of clinical trials (eg, using NAD+ precursors). The meeting facilitated dialogue among policymakers, administrative leaders, researchers, and clinical experts, aiming to promote international research collaborations and to translate findings into clinical applications and interventions to advance healthy aging.

Unhealthy aging poses a global challenge with profound healthcare and socioeconomic implications. Slowing down the aging process offers a promising approach to reduce the burden of a number of age-related diseases, such as dementia, and promoting healthy longevity in the old population. In response to the challenge of the aging population and with a view to the future, Norway and the United Kingdom are fostering collaborations, supported by a "Money Follows Cooperation agreement" between the 2 nations. The inaugural Norway-UK joint meeting on aging and dementia gathered leading experts on aging and dementia from the 2 nations to share their latest discoveries in related fields. Since aging is an international challenge, and to foster collaborations, we also invited leading scholars from 11 additional countries to join this event. This report provides a summary of the conference, highlighting recent progress on molecular aging mechanisms, genetic risk factors, DNA damage and repair, mitophagy, autophagy, as well as progress on a series of clinical trials (eg, using NAD+ precursors). The meeting facilitated dialogue among policymakers, administrative leaders, researchers, and clinical experts, aiming to promote international research collaborations and to translate findings into clinical applications and interventions to advance healthy aging.

Unhealthy aging poses a global challenge with profound healthcare and socioeconomic implications. Slowing down the aging process offers a promising approach to reduce the burden of a number of age-related diseases, such as dementia, and promoting healthy longevity in the old population. In response to the challenge of the aging population and with a view to the future, Norway and the United Kingdom are fostering collaborations, supported by a “Money Follows Cooperation agreement” between the 2 nations. The inaugural Norway–UK joint meeting on aging and dementia gathered leading experts on aging and dementia from the 2 nations to share their latest discoveries in related fields. Since aging is an international challenge, and to foster collaborations, we also invited leading scholars from 11 additional countries to join this event. This report provides a summary of the conference, highlighting recent progress on molecular aging mechanisms, genetic risk factors, DNA damage and repair, mitophagy, autophagy, as well as progress on a series of clinical trials (eg, using NAD+ precursors). The meeting facilitated dialogue among policymakers, administrative leaders, researchers, and clinical experts, aiming to promote international research collaborations and to translate findings into clinical applications and interventions to advance healthy aging., Unhealthy aging poses a global challenge with profound healthcare and socioeconomic implications. Slowing down the aging process offers a promising approach to reduce the burden of a number of age-related diseases, such as dementia, and promoting healthy longevity in the old population. In response to the challenge of the aging population and with a view to the future, Norway and the United Kingdom are fostering collaborations, supported by a "Money Follows Cooperation agreement" between the 2 nations. The inaugural Norway-UK joint meeting on aging and dementia gathered leading experts on aging and dementia from the 2 nations to share their latest discoveries in related fields. Since aging is an international challenge, and to foster collaborations, we also invited leading scholars from 11 additional countries to join this event. This report provides a summary of the conference, highlighting recent progress on molecular aging mechanisms, genetic risk factors, DNA damage and repair, mitophagy, autophagy, as well as progress on a series of clinical trials (eg, using NAD+ precursors). The meeting facilitated dialogue among policymakers, administrative leaders, researchers, and clinical experts, aiming to promote international research collaborations and to translate findings into clinical applications and interventions to advance healthy aging.

Channel charting, an unsupervised learning method that learns a low-dimensional representation from channel information to preserve geometrical property of physical space of user equipments (UEs), has drawn many attentions from both academic and industrial communities, because it can facilitate many downstream tasks, such as indoor localization, UE handover, beam management, and so on. However, many previous works mainly focus on charting that only preserves local geometry and use raw channel information to learn the chart, which do not consider the global geometry and are often computationally intensive and very time-consuming. Therefore, in this paper, a novel signature based approach for global channel charting with ultra low complexity is proposed. By using an iterated-integral based method called signature transform, a compact feature map and a novel distance metric are proposed, which enable channel charting with ultra low complexity and preserving both local and global geometry. We demonstrate the efficacy of our method using synthetic and open-source real-field datasets., Comment: accepted by IEEE ICC 2024 Workshops

We investigate a new task in human motion prediction, which is predicting motions under unexpected physical perturbation potentially involving multiple people. Compared with existing research, this task involves predicting less controlled, unpremeditated and pure reactive motions in response to external impact and how such motions can propagate through people. It brings new challenges such as data scarcity and predicting complex interactions. To this end, we propose a new method capitalizing differential physics and deep neural networks, leading to an explicit Latent Differential Physics (LDP) model. Through experiments, we demonstrate that LDP has high data efficiency, outstanding prediction accuracy, strong generalizability and good explainability. Since there is no similar research, a comprehensive comparison with 11 adapted baselines from several relevant domains is conducted, showing LDP outperforming existing research both quantitatively and qualitatively, improving prediction accuracy by as much as 70%, and demonstrating significantly stronger generalization.

In offline reinforcement learning (RL), the absence of active exploration calls for attention on the model robustness to tackle the sim-to-real gap, where the discrepancy between the simulated and deployed environments can significantly undermine the performance of the learned policy. To endow the learned policy with robustness in a sample-efficient manner in the presence of high-dimensional state-action space, this paper considers the sample complexity of distributionally robust linear Markov decision processes (MDPs) with an uncertainty set characterized by the total variation distance using offline data. We develop a pessimistic model-based algorithm and establish its sample complexity bound under minimal data coverage assumptions, which outperforms prior art by at least $\tilde{O}(d)$, where $d$ is the feature dimension. We further improve the performance guarantee of the proposed algorithm by incorporating a carefully-designed variance estimator., Comment: under review

Pedestrian trajectory prediction is the key technology in many applications for providing insights into human behavior and anticipating human future motions. Most existing empirical models are explicitly formulated by observed human behaviors using explicable mathematical terms with a deterministic nature, while recent work has focused on developing hybrid models combined with learning-based techniques for powerful expressiveness while maintaining explainability. However, the deterministic nature of the learned steering behaviors from the empirical models limits the models' practical performance. To address this issue, this work proposes the social conditional variational autoencoder (SocialCVAE) for predicting pedestrian trajectories, which employs a CVAE to explore behavioral uncertainty in human motion decisions. SocialCVAE learns socially reasonable motion randomness by utilizing a socially explainable interaction energy map as the CVAE's condition, which illustrates the future occupancy of each pedestrian's local neighborhood area. The energy map is generated using an energy-based interaction model, which anticipates the energy cost (i.e., repulsion intensity) of pedestrians' interactions with neighbors. Experimental results on two public benchmarks including 25 scenes demonstrate that SocialCVAE significantly improves prediction accuracy compared with the state-of-the-art methods, with up to 16.85% improvement in Average Displacement Error (ADE) and 69.18% improvement in Final Displacement Error (FDE)., Comment: Accepted by AAAI'24

This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated evaluation benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding., Comment: Project page: https://qizekun.github.io/shapellm

We propose a new method for cloth digitalization. Deviating from existing methods which learn from data captured under relatively casual settings, we propose to learn from data captured in strictly tested measuring protocols, and find plausible physical parameters of the cloths. However, such data is currently absent, so we first propose a new dataset with accurate cloth measurements. Further, the data size is considerably smaller than the ones in current deep learning, due to the nature of the data capture process. To learn from small data, we propose a new Bayesian differentiable cloth model to estimate the complex material heterogeneity of real cloths. It can provide highly accurate digitalization from very limited data samples. Through exhaustive evaluation and comparison, we show our method is accurate in cloth digitalization, efficient in learning from limited data samples, and general in capturing material variations. Code and data are available https://github.com/realcrane/Bayesian-Differentiable-Physics-for-Cloth-Digitalization, Comment: 9 pages, 8 figures, to be published in CVPR

Vision-and-language navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavor to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometers, or depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision making and instruction following. We train NaVid with 510k navigation samples collected from continuous environments, including action-planning and instruction-reasoning samples, along with 763k large-scale web data. Extensive experiments show that NaVid achieves state-of-the-art performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field., Comment: Accepted by Robotics: Science and Systems (RSS 2024)

Accurate pedestrian intention prediction (PIP) by Autonomous Vehicles (AVs) is one of the current research challenges in this field. In this article, we introduce PIP-Net, a novel framework designed to predict pedestrian crossing intentions by AVs in real-world urban scenarios. We offer two variants of PIP-Net designed for different camera mounts and setups. Leveraging both kinematic data and spatial features from the driving scene, the proposed model employs a recurrent and temporal attention-based solution, outperforming state-of-the-art performance. To enhance the visual representation of road users and their proximity to the ego vehicle, we introduce a categorical depth feature map, combined with a local motion flow feature, providing rich insights into the scene dynamics. Additionally, we explore the impact of expanding the camera's field of view, from one to three cameras surrounding the ego vehicle, leading to enhancement in the model's contextual perception. Depending on the traffic scenario and road environment, the model excels in predicting pedestrian crossing intentions up to 4 seconds in advance which is a breakthrough in current research studies in pedestrian intention prediction. Finally, for the first time, we present the Urban-PIP dataset, a customised pedestrian intention prediction dataset, with multi-camera annotations in real-world automated driving scenarios.

Dynamic magnetic resonance imaging (MRI) plays an indispensable role in cardiac diagnosis. To enable fast imaging, the k-space data can be undersampled but the image reconstruction poses a great challenge of high-dimensional processing. This challenge leads to necessitate extensive training data in many deep learning reconstruction methods. This work proposes a novel and efficient approach, leveraging a dimension-reduced separable learning scheme that excels even with highly limited training data. We further integrate it with spatiotemporal priors to develop a Deep Separable Spatiotemporal Learning network (DeepSSL), which unrolls an iteration process of a reconstruction model with both temporal low-rankness and spatial sparsity. Intermediate outputs are visualized to provide insights into the network's behavior and enhance its interpretability. Extensive results on cardiac cine datasets show that the proposed DeepSSL is superior to the state-of-the-art methods visually and quantitatively, while reducing the demand for training cases by up to 75%. And its preliminary adaptability to cardiac patients has been verified through experienced radiologists' and cardiologists' blind reader study. Additionally, DeepSSL also benefits for achieving the downstream task of cardiac segmentation with higher accuracy and shows robustness in prospective real-time cardiac MRI., Comment: 10 pages, 11 figures, 3 tables

Data generation is recognized as a potent strategy for unsupervised domain adaptation (UDA) pertaining semantic segmentation in adverse weathers. Nevertheless, these adverse weather scenarios encompass multiple possibilities, and high-fidelity data synthesis with controllable weather is under-researched in previous UDA works. The recent strides in large-scale text-to-image diffusion models (DM) have ushered in a novel avenue for research, enabling the generation of realistic images conditioned on semantic labels. This capability proves instrumental for cross-domain data synthesis from source to target domain owing to their shared label space. Thus, source domain labels can be paired with those generated pseudo target data for training UDA. However, from the UDA perspective, there exists several challenges for DM training: (i) ground-truth labels from target domain are missing; (ii) the prompt generator may produce vague or noisy descriptions of images from adverse weathers; (iii) existing arts often struggle to well handle the complex scene structure and geometry of urban scenes when conditioned only on semantic labels. To tackle the above issues, we propose ControlUDA, a diffusion-assisted framework tailored for UDA segmentation under adverse weather conditions. It first leverages target prior from a pre-trained segmentor for tuning the DM, compensating the missing target domain labels; It also contains UDAControlNet, a condition-fused multi-scale and prompt-enhanced network targeted at high-fidelity data generation in adverse weathers. Training UDA with our generated data brings the model performances to a new milestone (72.0 mIoU) on the popular Cityscapes-to-ACDC benchmark for adverse weathers. Furthermore, ControlUDA helps to achieve good model generalizability on unseen data.

The widespread adoption of third-party libraries (TPLs) in software development has accelerated the creation of modern software. However, this convenience comes with potential legal risks. Developers may inadvertently violate the licenses of TPLs, leading to legal issues. While existing studies have explored software licenses and potential incompatibilities, these studies often focus on a limited set of licenses or rely on low-quality license data, which may affect their conclusions. To address this gap, there is a need for a high-quality license dataset that encompasses a broad range of mainstream licenses to help developers navigate the complex landscape of software licenses, avoid potential legal pitfalls, and guide solutions for managing license compliance and compatibility in software development. To this end, we conduct the first work to understand the mainstream software licenses based on term granularity and obtain a high-quality dataset of 453 SPDX licenses with well-labeled terms and conflicts. Specifically, we first conduct a differential analysis of the mainstream platforms to understand the terms and attitudes of each license. Next, we propose a standardized set of license terms to capture and label existing mainstream licenses with high quality. Moreover, we include copyleft conflicts and conclude the three major types of license conflicts among the 453 SPDX licenses. Based on these, we carry out two empirical studies to reveal the concerns and threats from the perspectives of both licensors and licensees. One study provides an in-depth analysis of the similarities, differences, and conflicts among SPDX licenses, revisits the usage and conflicts of licenses in the NPM ecosystem, and draws conclusions that differ from previous work. Our studies reveal some insightful findings and disclose relevant analytical data, which set the stage for further research., Comment: 10 pages, 6 figures, accepted by SANER2024

Open-vocabulary 3D instance segmentation is cutting-edge for its ability to segment 3D instances without predefined categories. However, progress in 3D lags behind its 2D counterpart due to limited annotated 3D data. To address this, recent works first generate 2D open-vocabulary masks through 2D models and then merge them into 3D instances based on metrics calculated between two neighboring frames. In contrast to these local metrics, we propose a novel metric, view consensus rate, to enhance the utilization of multi-view observations. The key insight is that two 2D masks should be deemed part of the same 3D instance if a significant number of other 2D masks from different views contain both these two masks. Using this metric as edge weight, we construct a global mask graph where each mask is a node. Through iterative clustering of masks showing high view consensus, we generate a series of clusters, each representing a distinct 3D instance. Notably, our model is training-free. Through extensive experiments on publicly available datasets, including ScanNet++, ScanNet200 and MatterPort3D, we demonstrate that our method achieves state-of-the-art performance in open-vocabulary 3D instance segmentation. Our project page is at https://pku-epic.github.io/MaskClustering.

This paper describes our audio-quality-based multi-strategy approach for the audio-visual target speaker extraction (AVTSE) task in the Multi-modal Information based Speech Processing (MISP) 2023 Challenge. Specifically, our approach adopts different extraction strategies based on the audio quality, striking a balance between interference removal and speech preservation, which benifits the back-end automatic speech recognition (ASR) systems. Experiments show that our approach achieves a character error rate (CER) of 24.2% and 33.2% on the Dev and Eval set, respectively, obtaining the second place in the challenge., Comment: Accepted by ICASSP 2024

The emergence of neural networks constrained by physical governing equations has sparked a new trend in deep learning research, which is known as Physics-Informed Neural Networks (PINNs). However, solving high-dimensional problems with PINNs is still a substantial challenge, the space complexity brings difficulty to solving large multidirectional problems. In this paper, a novel PINN framework to quickly predict several three-dimensional Terzaghi consolidation cases under different conditions is proposed. Meanwhile, the loss functions for different cases are introduced, and their differences in three-dimensional consolidation problems are highlighted. The tuning strategies for the PINNs framework for three-dimensional consolidation problems are introduced. Then, the performance of PINNs is tested and compared with traditional numerical methods adopted in forward problems, and the coefficients of consolidation and the impact of noisy data in inverse problems are identified. Finally, the results are summarized and presented from three-dimensional simulations of PINNs, which show an accuracy rate of over 99% compared with ground truth for both forward and inverse problems. These results are desirable with good accuracy and can be used for soil settlement prediction, which demonstrates that the proposed PINNs framework can learn the three-dimensional consolidation PDE well. Keywords: Three-dimensional Terzaghi consolidation; Physics-informed neural networks (PINNs); Forward problems; Inverse problems; soil settlement, Comment: 30 pages, 11 figures, 6 tables, 23 equations

This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task. In terms of data processing, we leverage the lip motion extractor from the baseline1 to produce multi-scale video data. Besides, various augmentation techniques are applied during training, encompassing speed perturbation, random rotation, horizontal flipping, and color transformation. The VSR model adopts an end-to-end architecture with joint CTC/attention loss, comprising a ResNet3D visual frontend, an E-Branchformer encoder, and a Transformer decoder. Experiments show that our system achieves 34.76% CER for the Single-Speaker Task and 41.06% CER for the Multi-Speaker Task after multi-system fusion, ranking first place in all three tracks we participate., Comment: Included in CNVSRC Workshop 2023, NCMMSC 2023

To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge. This challenge collects over 100 hours of multi-channel speech data recorded inside a new energy vehicle and 40 hours of noise for data augmentation. Two tracks, including automatic speech recognition (ASR) and automatic speech diarization and recognition (ASDR) are set up, using character error rate (CER) and concatenated minimum permutation character error rate (cpCER) as evaluation metrics, respectively. Overall, the ICMC-ASR Challenge attracts 98 participating teams and receives 53 valid results in both tracks. In the end, first-place team USTCiflytek achieves a CER of 13.16% in the ASR track and a cpCER of 21.48% in the ASDR track, showing an absolute improvement of 13.08% and 51.4% compared to our challenge baseline, respectively., Comment: Accepted at ICASSP 2024

While automatic speech recognition (ASR) systems degrade significantly in noisy environments, audio-visual speech recognition (AVSR) systems aim to complement the audio stream with noise-invariant visual cues and improve the system's robustness. However, current studies mainly focus on fusing the well-learned modality features, like the output of modality-specific encoders, without considering the contextual relationship during the modality feature learning. In this study, we propose a multi-layer cross-attention fusion based AVSR (MLCA-AVSR) approach that promotes representation learning of each modality by fusing them at different levels of audio/visual encoders. Experimental results on the MISP2022-AVSR Challenge dataset show the efficacy of our proposed system, achieving a concatenated minimum permutation character error rate (cpCER) of 30.57% on the Eval set and yielding up to 3.17% relative improvement compared with our previous system which ranked the second place in the challenge. Following the fusion of multiple systems, our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset., Comment: 5 pages, 3 figures Accepted at ICASSP 2024

In cardiac Magnetic Resonance Imaging (MRI) analysis, simultaneous myocardial segmentation and T2 quantification are crucial for assessing myocardial pathologies. Existing methods often address these tasks separately, limiting their synergistic potential. To address this, we propose SQNet, a dual-task network integrating Transformer and Convolutional Neural Network (CNN) components. SQNet features a T2-refine fusion decoder for quantitative analysis, leveraging global features from the Transformer, and a segmentation decoder with multiple local region supervision for enhanced accuracy. A tight coupling module aligns and fuses CNN and Transformer branch features, enabling SQNet to focus on myocardium regions. Evaluation on healthy controls (HC) and acute myocardial infarction patients (AMI) demonstrates superior segmentation dice scores (89.3/89.2) compared to state-of-the-art methods (87.7/87.9). T2 quantification yields strong linear correlations (Pearson coefficients: 0.84/0.93) with label values for HC/AMI, indicating accurate mapping. Radiologist evaluations confirm SQNet's superior image quality scores (4.60/4.58 for segmentation, 4.32/4.42 for T2 quantification) over state-of-the-art methods (4.50/4.44 for segmentation, 3.59/4.37 for T2 quantification). SQNet thus offers accurate simultaneous segmentation and quantification, enhancing cardiac disease diagnosis, such as AMI., Comment: 10 pages, 8 figures, 6 tables

We present a new deep learning paradigm for the generation of sparse approximate inverse (SPAI) preconditioners for matrix systems arising from the mesh-based discretization of elliptic differential operators. Our approach is based upon the observation that matrices generated in this manner are not arbitrary, but inherit properties from differential operators that they discretize. Consequently, we seek to represent a learnable distribution of high-performance preconditioners from a low-dimensional subspace through a carefully-designed autoencoder, which is able to generate SPAI preconditioners for these systems. The concept has been implemented on a variety of finite element discretizations of second- and fourth-order elliptic partial differential equations with highly promising results., Comment: 15 pages, 8 figures, International conference on Computational Science

Unlike the classical distributed consensus protocols enabling the group of agents as a whole to reach an agreement regarding a certain quantity of interest in a distributed fashion, the distributed biased min-consensus protocol (DBMC) has been proven to generate advanced complexity pertaining to solving the shortest path problem. As such a protocol is commonly incorporated as the first step of a hierarchical architecture in real applications, e.g., robots path planning, management of dispersed computing services, an impedance limiting the application potential of DBMC lies in, the lack of results regarding to its convergence within a user-assigned time. In this paper, we first propose two control strategies ensuring the state error of DBMC decrease exactly to zero or a desired level manipulated by the user, respectively. To compensate the high feedback gains incurred by these two control strategies, this paper further investigates the nominal DBMC itself. By leveraging small gain based stability tools, this paper also proves the global exponential input-to-state stability of DBMC, outperforming its current stability results. Simulations have been provided to validate the efficacy of our theoretical result.

Automatic lip-reading (ALR) aims to automatically transcribe spoken content from a speaker's silent lip motion captured in video. Current mainstream lip-reading approaches only use a single visual encoder to model input videos of a single scale. In this paper, we propose to enhance lip-reading by incorporating multi-scale video data and multi-encoder. Specifically, we first propose a novel multi-scale lip motion extraction algorithm based on the size of the speaker's face and an Enhanced ResNet3D visual front-end (VFE) to extract lip features at different scales. For the multi-encoder, in addition to the mainstream Transformer and Conformer, we also incorporate the recently proposed Branchformer and E-Branchformer as visual encoders. In the experiments, we explore the influence of different video data scales and encoders on ALR system performance and fuse the texts transcribed by all ALR systems using recognizer output voting error reduction (ROVER). Finally, our proposed approach placed second in the ICME 2024 ChatCLR Challenge Task 2, with a 21.52% reduction in character error rate (CER) compared to the official baseline on the evaluation set., Comment: 6 pages, 3 figures, Accepted at ICMEW 2024

Close and continuous interaction with rich contacts is a crucial aspect of human activities (e.g. hugging, dancing) and of interest in many domains like activity recognition, motion prediction, character animation, etc. However, acquiring such skeletal motion is challenging. While direct motion capture is expensive and slow, motion editing/generation is also non-trivial, as complex contact patterns with topological and geometric constraints have to be retained. To this end, we propose a new deep learning method for two-body skeletal interaction motion augmentation, which can generate variations of contact-rich interactions with varying body sizes and proportions while retaining the key geometric/topological relations between two bodies. Our system can learn effectively from a relatively small amount of data and generalize to drastically different skeleton sizes. Through exhaustive evaluation and comparison, we show it can generate high-quality motions, has strong generalizability and outperforms traditional optimization-based methods and alternative deep learning solutions.

In this paper, we tackle the problem of grasping transparent and specular objects. This issue holds importance, yet it remains unsolved within the field of robotics due to failure of recover their accurate geometry by depth cameras. For the first time, we propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. ASGrasp utilizes a two-layer learning-based stereo network for the purpose of transparent object reconstruction, enabling material-agnostic object grasping in cluttered environments. In contrast to existing RGB-D based grasp detection methods, which heavily depend on depth restoration networks and the quality of depth maps generated by depth cameras, our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction. We create an extensive synthetic dataset through domain randomization, which is based on GraspNet-1Billion. Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping in both simulation and the real via seamless sim-to-real transfer. Our method significantly outperforms SOTA networks and even surpasses the performance upper bound set by perfect visible point cloud inputs.Project page: https://pku-epic.github.io/ASGrasp, Comment: IEEE International Conference on Robotics and Automation (ICRA), 2024

Large Language Models (LLMs) have demonstrated unparalleled effectiveness in various NLP tasks, and integrating LLMs with automatic speech recognition (ASR) is becoming a mainstream paradigm. Building upon this momentum, our research delves into an in-depth examination of this paradigm on a large open-source Chinese dataset. Specifically, our research aims to evaluate the impact of various configurations of speech encoders, LLMs, and projector modules in the context of the speech foundation encoder-LLM ASR paradigm. Furthermore, we introduce a three-stage training approach, expressly developed to enhance the model's ability to align auditory and textual information. The implementation of this approach, alongside the strategic integration of ASR components, enabled us to achieve the SOTA performance on the AISHELL-1, Test_Net, and Test_Meeting test sets. Our analysis presents an empirical foundation for future research in LLM-based ASR systems and offers insights into optimizing performance using Chinese datasets. We will publicly release all scripts used for data preparation, training, inference, and scoring, as well as pre-trained models and training logs to promote reproducible research.

RGBT tracking draws increasing attention due to its robustness in multi-modality warranting (MMW) scenarios, such as nighttime and bad weather, where relying on a single sensing modality fails to ensure stable tracking results. However, the existing benchmarks predominantly consist of videos collected in common scenarios where both RGB and thermal infrared (TIR) information are of sufficient quality. This makes the data unrepresentative of severe imaging conditions, leading to tracking failures in MMW scenarios. To bridge this gap, we present a new benchmark, MV-RGBT, captured specifically in MMW scenarios. In contrast with the existing datasets, MV-RGBT comprises more object categories and scenes, providing a diverse and challenging benchmark. Furthermore, for severe imaging conditions of MMW scenarios, a new problem is posed, namely \textit{when to fuse}, to stimulate the development of fusion strategies for such data. We propose a new method based on a mixture of experts, namely MoETrack, as a baseline fusion strategy. In MoETrack, each expert generates independent tracking results along with the corresponding confidence score, which is used to control the fusion process. Extensive experimental results demonstrate the significant potential of MV-RGBT in advancing RGBT tracking and elicit the conclusion that fusion is not always beneficial, especially in MMW scenarios. Significantly, the proposed MoETrack method achieves new state-of-the-art results not only on MV-RGBT, but also on standard benchmarks, such as RGBT234, LasHeR, and the short-term split of VTUAV (VTUAV-ST). More information of MV-RGBT and the source code of MoETrack will be released at https://github.com/Zhangyong-Tang/MoETrack.

In the burgeoning field of Large Language Models (LLMs) like ChatGPT and LLaMA, Prompt Engineering (PE) is renowned for boosting zero-shot or in-context learning (ICL) through prompt modifications. Yet, the realm of the sample design for downstream fine-tuning, crucial for task-specific LLM adaptation, is largely unexplored. This paper introduces Sample Design Engineering (SDE), a methodical approach to enhancing LLMs' post-tuning performance by refining input, output, and reasoning designs. We conduct a series of in-domain (ID) and out-of-domain (OOD) experiments to assess the impact of various design options on LLMs' downstream performance, revealing several intriguing patterns that hold consistently across different LLMs. Based on these insights, we propose an integrated SDE strategy, combining the most effective options, and validate its consistent superiority over heuristic sample designs in complex downstream tasks like multi-aspect sentiment analysis, event extraction, and nested entity recognition. Additionally, analyses of LLMs' inherent prompt/output perplexity, zero-shot, and ICL abilities illustrate that good PE strategies may not always translate to good SDE strategies. Code available at https://github.com/beyondguo/LLM-Tuning., Comment: 23 pages, 12 figures, 14 tables

Physics-informed neural networks (PINNs) provide a means of obtaining approximate solutions of partial differential equations and systems through the minimisation of an objective function which includes the evaluation of a residual function at a set of collocation points within the domain. The quality of a PINNs solution depends upon numerous parameters, including the number and distribution of these collocation points. In this paper we consider a number of strategies for selecting these points and investigate their impact on the overall accuracy of the method. In particular, we suggest that no single approach is likely to be ``optimal'' but we show how a number of important metrics can have an impact in improving the quality of the results obtained when using a fixed number of residual evaluations. We illustrate these approaches through the use of two benchmark test problems: Burgers' equation and the Allen-Cahn equation., Comment: 15 pages, 8 figures, 2 tables. Accepted for publication in the conference proceedings of the International Conference on Computational Science (ICCS) 2024

To date, many experiments have revealed that the functional balance between hemagglutinin (HA) and neuraminidase (NA) plays a crucial role in viral mobility, production, and transmission. However, whether and how HA and NA maintain balance at the sequence level needs further investigation. Here, we applied principal component analysis and hierarchical clustering analysis on thousands of HA and NA sequences of A/H1N1 and A/H3N2. We discovered significant coevolution between HA and NA at the sequence level, which is closely related to the type of host species and virus epidemic years. Furthermore, we propose a sequence-to-sequence transformer model (S2STM), which mainly consists of an encoder and a decoder that adopts a multi-head attention mechanism for establishing the mapping relationship between HA and NA sequences. The training results reveal that the S2STM can effectively realize the "translation" from HA to NA or vice versa, thereby building a relationship network between them. Our work combines unsupervised and supervised machine learning methods to identify the sequence matching between HA and NA, which will advance our understanding of IAVs' evolution and also provide a novel idea for sequence analysis methods.

To date, many experiments have revealed that the functional balance between hemagglutinin (HA) and neuraminidase (NA) plays a crucial role in viral mobility, production, and transmission. However, whether and how HA and NA maintain balance at the sequence level needs further investigation. Here, we applied principal component analysis and hierarchical clustering analysis on thousands of HA and NA sequences of A/H1N1 and A/H3N2. We discovered significant coevolution between HA and NA at the sequence level, which is closely related to the type of host species and virus epidemic years. Furthermore, we propose a sequence-to-sequence transformer model (S2STM), which mainly consists of an encoder and a decoder that adopts a multi-head attention mechanism for establishing the mapping relationship between HA and NA sequences. The training results reveal that the S2STM can effectively realize the "translation" from HA to NA or vice versa, thereby building a relationship network between them. Our work combines unsupervised and supervised machine learning methods to identify the sequence matching between HA and NA, which will advance our understanding of IAVs' evolution and also provide a novel idea for sequence analysis methods.

Changxing Liu,1,* Xinyi Guo,2,* Yabin Zhou,3 He Wang3 1First Clinical Medical School, Heilongjiang University of Chinese Medicine, Harbin, 150040, People’s Republic of China; 2Clinical Medical School, Chengdu University of Traditional Chinese Medicine, Chengdu, 610075, People’s Republic of China; 3Department of Cardiology, The First Hospital of Heilongjiang University of Chinese Medicine, Harbin, 150040, People’s Republic of China*These authors contributed equally to this workCorrespondence: Yabin Zhou; He Wang, Email yabzhou@163.com; hehe0808032@163.comAbstract: Heart failure (HF) is a complex clinical syndrome that represents the advanced stage of cardiovascular disease, characterized by systolic and diastolic dysfunction of the heart. Despite continuous updates in HF treatment drugs, the morbidity and mortality rates remain high, necessitating ongoing exploration for new therapeutic targets. Adenosine monophosphate-activated protein kinase (AMPK) is the serine/threonine protein kinase which responds to adenosine monophosphate (AMP) levels.Activation of AMPK shifts cellular metabolic patterns from synthesis to catabolism, enhancing energy metabolism in pathological conditions such as inflammation, ischemia, obesity, and aging. Numerous studies have identified AMPK as a vital target for HF treatment, with herbal monomers/extracts and compounds affecting key signaling factors including rapamycin targeting protein (mTOR), silencing regulator protein 1 (SIRT1), nuclear transcription factor E2-related factor 2 (Nrf2), and nuclear transcription factor-κB (NF-κB) through regulation of the AMPK signaling pathway.This modulation can achieve the effects of improving metabolism, autophagy, reducing oxidative stress and inflammatory response in the treatment of heart failure, with the advantages of multi-targeting, comprehensive action and low toxicity.The modulation of the AMPK pathway by Traditional Chinese Medicine (TCM) has emerged as a cru

Lianjun Lin,1,* Ruixue Zhang,1,* Zhi Zhang,2,3 Yujun Chang,2,3 Rongnan Lin,2,3 Haiwei Dou,4 He Wang,5 Yuchuan Wang,6 Bo Zheng7 1Geriatric Department, Peking University First Hospital, Beijing, People’s Republic of China; 2National Engineering Research Center for Beijing Biochip Technology, Beijing, People’s Republic of China; 3CapitalBio Corporation, Beijing, People’s Republic of China; 4Beijing Friendship Hospital, Capital Medical University, Beijing Tropical Medicine Research Institute, Beijing, People’s Republic of China; 5Department of Imaging, Peking University First Hospital, Beijing, People’s Republic of China; 6Department of Cardiology, Beijing Shijitan Hospital, Beijing, People’s Republic of China; 7Institute of Clinical Pharmacology, Peking University First Hospital, Beijing, People’s Republic of China*These authors contributed equally to this workCorrespondence: Zhi Zhang, National Engineering Research Center for Beijing Biochip Technology, Changping District, Beijing, 102206, People’s Republic of China, Email zhizhang@capitalbio.com Lianjun Lin, Geriatric Department, Peking University First Hospital, Xishiku Avenue 8, Xicheng District, Beijing, 100034, People’s Republic of China, Email 06474@pkufh.comBackground: Mycoplasma pneumoniae is an important infectious pathogen of lower respiratory tract infection in children and adolescents. Macrolide resistant M. pneumoniae (MRMP) has become increasingly prevalent, and identifying pathogen resistance genes is crucial for treatment.Case Presentation: We report a patient with severe refractory M. pneumoniae pneumonia (MPP). The failure of initial clinical treatment prompted the re-analysis of metagenomic next-generation sequencing (mNGS) data for macrolide-resistant gene. Macrolide-resistance 23S ribosomal RNA gene was confirmed with read depth of 64 X for the A2063G mutation, which can decrease the affinity of macrolide with M. pneumoniae ribosome resulting in macrolide

We present the results of the first Machine Learning Gravitational-Wave Search Mock Data Challenge. For this challenge, participating groups had to identify gravitational-wave signals from binary black hole mergers of increasing complexity and duration embedded in progressively more realistic noise. The final of the 4 provided datasets contained real noise from the O3a observing run and signals up to a duration of 20 s with the inclusion of precession effects and higher order modes. We present the average sensitivity distance and run-time for the 6 entered algorithms derived from 1 month of test data unknown to the participants prior to submission. Of these, 4 are machine learning algorithms. We find that the best machine learning based algorithms are able to achieve up to 95% of the sensitive distance of matched-filtering based production analyses for simulated Gaussian noise at a false-alarm rate (FAR) of one per month. In contrast, for real noise, the leading machine learning search achieved 70%. For higher FARs the differences in sensitive distance shrink to the point where select machine learning submissions outperform traditional search algorithms at FARs ≥200 per month on some datasets. Our results show that current machine learning search algorithms may already be sensitive enough in limited parameter regions to be useful for some production settings. To improve the state-of-the-art, machine learning algorithms need to reduce the false-alarm rates at which they are capable of detecting signals and extend their validity to regions of parameter space where modeled searches are computationally expensive to run. Based on our findings we compile a list of research areas that we believe are the most important to elevate machine learning searches to an invaluable tool in gravitational-wave signal detection.

Actor-critic (AC) cooperative multiagent reinforcement learning (MARL) over directed graphs is studied in this article. The goal of the agents in MARL is to maximize the globally averaged return in a distributed way, i.e., each agent can only exchange information with its neighboring agents. AC methods proposed in the literature require the communication graphs to be undirected and the weight matrices to be doubly stochastic (more precisely, the weight matrices are row stochastic and their expectation are column stochastic). Differently from these methods, we propose a distributed AC algorithm for MARL over directed graph with fixed topology that only requires the weight matrix to be row stochastic. Then, we also study the MARL over directed graphs (possibly not connected) with changing topologies, proposing a different distributed AC algorithm based on the push-sum protocol that only requires the weight matrices to be column stochastic. Convergence of the proposed algorithms is proven for linear function approximation of the action value function. Simulations are presented to demonstrate the effectiveness of the proposed algorithms., Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Team Bart De Schutter

Elucidating the cellular organization of the cerebral cortex is critical for understanding brain structure and function. Using large-scale single-nucleus RNA sequencing and spatial transcriptomic analysis of 143 macaque cortical regions, we obtained a comprehensive atlas of 264 transcriptome-defined cortical cell types and mapped their spatial distribution across the entire cortex. We characterized the cortical layer and region preferences of glutamatergic, GABAergic, and non-neuronal cell types, as well as regional differences in cell-type composition and neighborhood complexity. Notably, we discovered a relationship between the regional distribution of various cell types and the region’s hierarchical level in the visual and somatosensory systems. Cross-species comparison of transcriptomic data from human, macaque, and mouse cortices further revealed primate-specific cell types that are enriched in layer 4, with their marker genes expressed in a region-dependent manner. Our data provide a cellular and molecular basis for understanding the evolution, development, aging, and pathogenesis of the primate brain., Elucidating the cellular organization of the cerebral cortex is critical for understanding brain structure and function. Using large-scale single-nucleus RNA sequencing and spatial transcriptomic analysis of 143 macaque cortical regions, we obtained a comprehensive atlas of 264 transcriptome-defined cortical cell types and mapped their spatial distribution across the entire cortex. We characterized the cortical layer and region preferences of glutamatergic, GABAergic, and non-neuronal cell types, as well as regional differences in cell-type composition and neighborhood complexity. Notably, we discovered a relationship between the regional distribution of various cell types and the region's hierarchical level in the visual and somatosensory systems. Cross-species comparison of transcriptomic data from human, macaque, and mouse cortices further revealed primate-specific cell types that are enriched in layer 4, with their marker genes expressed in a region-dependent manner. Our data provide a cellular and molecular basis for understanding the evolution, development, aging, and pathogenesis of the primate brain.

Gravitational wave (GW) astronomy is witnessing a transformative shift from terrestrial to space-based detection, with missions like Taiji at the forefront. While the transition brings unprecedented opportunities for exploring massive black hole binaries (MBHBs), it also imposes complex challenges in data analysis, particularly in parameter estimation amidst confusion noise. Addressing this gap, we utilize scalable normalizing flow models to achieve rapid and accurate inference within the Taiji environment. Innovatively, our approach simplifies the data's complexity, employs a transformation mapping to overcome the year-period time-dependent response function, and unveils additional multimodality in the arrival time parameter. Our method estimates MBHBs several orders of magnitude faster than conventional techniques, maintaining high accuracy even in complex backgrounds. These findings significantly enhance the efficiency of GW data analysis, paving the way for rapid detection and alerting systems and enriching our ability to explore the universe through space-based GW observation., Comment: 14 pages, 7 figures. Published version

In the 2030s, a new era of gravitational-wave (GW) observations will dawn as multiple space-based GW detectors, such as the Laser Interferometer Space Antenna, Taiji and TianQin, open the millihertz window for GW astronomy. These detectors are poised to detect a multitude of GW signals emitted by different sources. It is a challenging task for GW data analysis to recover the parameters of these sources at a low computational cost. Generally, the matched filtering approach entails exploring an extensive parameter space for all resolvable sources, incurring a substantial cost owing to the generation of GW waveform templates. To alleviate the challenge, we make an attempt to perform parameter inference for coalescing massive black hole binaries (MBHBs) using deep learning. The model trained in this work has the capability to produce 50,000 posterior samples for redshifted total mass, mass ratio, coalescence time and luminosity distance of a MBHB in about twenty seconds. Our model can serve as a potent data pre-processing tool, reducing the volume of parameter space by more than four orders of magnitude for MBHB signals with a signal-to-noise ratio larger than 100. Moreover, the model exhibits robustness when handling input data that contains multiple MBHB signals., Comment: 8 pages, 4 figures

We investigated the dynamical behaviors of bimodular continuous attractor neural networks, each processing a modality of sensory input and interacting with each other. We found that when bumps coexist in both modules, the position of each bump is shifted towards the other input when the intermodular couplings are excitatory and is shifted away when inhibitory. When one intermodular coupling is excitatory while another is moderately inhibitory, temporally modulated population spikes can be generated. On further increase of the inhibitory coupling, momentary spikes will emerge. In the regime of bump coexistence, bump heights are primarily strengthened by excitatory intermodular couplings, but there is a lesser weakening effect due to a bump being displaced from the direct input. When bimodular networks serve as decoders of multisensory integration, we extend the Bayesian framework to show that excitatory and inhibitory couplings encode attractive and repulsive priors, respectively. At low disparity, the bump positions decode the posterior means in the Bayesian framework, whereas at high disparity, multiple steady states exist. In the regime of multiple steady states, the less stable state can be accessed if the input causing the more stable state arrives after a sufficiently long delay. When one input is moving, the bump in the corresponding module is pinned when the moving stimulus is weak, unpinned at intermediate stimulus strength, and tracks the input at strong stimulus strength, and the stimulus strengths for these transitions increase with the velocity of the moving stimulus. These results are important to understanding multisensory integration of static and dynamic stimuli.

Shape assembly composes complex shapes geometries by arranging simple part geometries and has wide applications in autonomous robotic assembly and CAD modeling. Existing works focus on geometry reasoning and neglect the actual physical assembly process of matching and fitting joints, which are the contact surfaces connecting different parts. In this paper, we consider contacting joints for the task of multi-part assembly. A successful joint-optimized assembly needs to satisfy the bilateral objectives of shape structure and joint alignment. We propose a hierarchical graph learning approach composed of two levels of graph representation learning. The part graph takes part geometries as input to build the desired shape structure. The joint-level graph uses part joints information and focuses on matching and aligning joints. The two kinds of information are combined to achieve the bilateral objectives. Extensive experiments demonstrate that our method outperforms previous methods, achieving better shape structure and higher joint alignment accuracy.

In this work, we tackle the problem of learning universal robotic dexterous grasping from a point cloud observation under a table-top setting. The goal is to grasp and lift up objects in high-quality and diverse ways and generalize across hundreds of categories and even the unseen. Inspired by successful pipelines used in parallel gripper grasping, we split the task into two stages: 1) grasp proposal (pose) generation and 2) goal-conditioned grasp execution. For the first stage, we propose a novel probabilistic model of grasp pose conditioned on the point cloud observation that factorizes rotation from translation and articulation. Trained on our synthesized large-scale dexterous grasp dataset, this model enables us to sample diverse and high-quality dexterous grasp poses for the object point cloud.For the second stage, we propose to replace the motion planning used in parallel gripper grasping with a goal-conditioned grasp policy, due to the complexity involved in dexterous grasping execution. Note that it is very challenging to learn this highly generalizable grasp policy that only takes realistic inputs without oracle states. We thus propose several important innovations, including state canonicalization, object curriculum, and teacher-student distillation. Integrating the two stages, our final pipeline becomes the first to achieve universal generalization for dexterous grasping, demonstrating an average success rate of more than 60\% on thousands of object instances, which significantly outperforms all baselines, meanwhile showing only a minimal generalization gap., Comment: Accepted to CVPR 2023

Estimating the 3DoF rotation from a single RGB image is an important yet challenging problem. Probabilistic rotation regression has raised more and more attention with the benefit of expressing uncertainty information along with the prediction. Though modeling noise using Gaussian-resembling Bingham distribution and matrix Fisher distribution is natural, they are shown to be sensitive to outliers for the nature of quadratic punishment to deviations. In this paper, we draw inspiration from multivariate Laplace distribution and propose a novel Rotation Laplace distribution on SO(3). Rotation Laplace distribution is robust to the disturbance of outliers and enforces much gradient to the low-error region, resulting in a better convergence. Our extensive experiments show that our proposed distribution achieves state-of-the-art performance for rotation regression tasks over both probabilistic and non-probabilistic baselines. Our project page is at https://pku-epic.github.io/RotationLaplace., Comment: ICLR 2023 Spotlight (notable-top-25%)

Category-level articulated object pose estimation aims to estimate a hierarchy of articulation-aware object poses of an unseen articulated object from a known category. To reduce the heavy annotations needed for supervised learning methods, we present a novel self-supervised strategy that solves this problem without any human labels. Our key idea is to factorize canonical shapes and articulated object poses from input articulated shapes through part-level equivariant shape analysis. Specifically, we first introduce the concept of part-level SE(3) equivariance and devise a network to learn features of such property. Then, through a carefully designed fine-grained pose-shape disentanglement strategy, we expect that canonical spaces to support pose estimation could be induced automatically. Thus, we could further predict articulated object poses as per-part rigid transformations describing how parts transform from their canonical part spaces to the camera space. Extensive experiments demonstrate the effectiveness of our method on both complete and partial point clouds from synthetic and real articulated object datasets., Comment: ICLR 2023

The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge., Comment: 5 pages. Accepted at ICASSP2023