Search

3D Gaussian Splatting (GS) demonstrates excellent rendering quality and generation speed in novel view synthesis. However, substantial data size poses challenges for storage and transmission, making 3D GS compression an essential technology. Current 3D GS compression research primarily focuses on developing more compact scene representations, such as converting explicit 3D GS data into implicit forms. In contrast, compression of the GS data itself has hardly been explored. To address this gap, we propose a Hierarchical GS Compression (HGSC) technique. Initially, we prune unimportant Gaussians based on importance scores derived from both global and local significance, effectively reducing redundancy while maintaining visual quality. An Octree structure is used to compress 3D positions. Based on the 3D GS Octree, we implement a hierarchical attribute compression strategy by employing a KD-tree to partition the 3D GS into multiple blocks. We apply farthest point sampling to select anchor primitives within each block and others as non-anchor primitives with varying Levels of Details (LoDs). Anchor primitives serve as reference points for predicting non-anchor primitives across different LoDs to reduce spatial redundancy. For anchor primitives, we use the region adaptive hierarchical transform to achieve near-lossless compression of various attributes. For non-anchor primitives, each is predicted based on the k-nearest anchor primitives. To further minimize prediction errors, the reconstructed LoD and anchor primitives are combined to form new anchor primitives to predict the next LoD. Our method notably achieves superior compression quality and a significant data size reduction of over 4.5 times compared to the state-of-the-art compression method on small scenes datasets.

In the current Video-based Dynamic Mesh Coding (V-DMC) standard, inter-frame coding is restricted to mesh frames with constant topology. Consequently, temporal redundancy is not fully leveraged, resulting in suboptimal compression efficacy. To address this limitation, this paper introduces a novel coarse-to-fine scheme to generate anchor meshes for frames with time-varying topology. Initially, we generate a coarse anchor mesh using an octree-based nearest neighbor search. Motion estimation compensates for regions with significant motion changes during this process. However, the quality of the coarse mesh is low due to its suboptimal vertices. To enhance details, the fine anchor mesh is further optimized using the Quadric Error Metrics (QEM) algorithm to calculate more precise anchor points. The inter-frame anchor mesh generated herein retains the connectivity of the reference base mesh, while concurrently preserving superior quality. Experimental results show that our method achieves 7.2% ~ 10.3% BD-rate gain compared to the existing V-DMC test model version 7.

Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting-where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.

We propose a novel end-to-end multi-talker automatic speech recognition (ASR) framework that enables both multi-speaker (MS) ASR and target-speaker (TS) ASR. Our proposed model is trained in a fully end-to-end manner, incorporating speaker supervision from a pre-trained speaker diarization module. We introduce an intuitive yet effective method for masking ASR encoder activations using output from the speaker supervision module, a technique we term Meta-Cat (meta-information concatenation), that can be applied to both MS-ASR and TS-ASR. Our results demonstrate that the proposed architecture achieves competitive performance in both MS-ASR and TS-ASR tasks, without the need for traditional methods, such as neural mask estimation or masking at the audio or feature level. Furthermore, we demonstrate a glimpse of a unified dual-task model which can efficiently handle both MS-ASR and TS-ASR tasks. Thus, this work illustrates that a robust end-to-end multi-talker ASR framework can be implemented with a streamlined architecture, obviating the need for the complex speaker filtering mechanisms employed in previous studies.

We propose Sortformer, a novel neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models. The permutation problem in speaker diarization has long been regarded as a critical challenge. Most prior end-to-end diarization systems employ permutation invariant loss (PIL), which optimizes for the permutation that yields the lowest error. In contrast, we introduce Sort Loss, which enables a diarization model to autonomously resolve permutation, with or without PIL. We demonstrate that combining Sort Loss and PIL achieves performance competitive with state-of-the-art end-to-end diarization models trained exclusively with PIL. Crucially, we present a streamlined multispeaker ASR architecture that leverages Sortformer as a speaker supervision model, embedding speaker label estimation within the ASR encoder state using a sinusoidal kernel function. This approach resolves the speaker permutation problem through sorted objectives, effectively bridging speaker-label timestamps and speaker tokens. In our experiments, we show that the proposed multispeaker ASR architecture, enhanced with speaker supervision, improves performance via adapter techniques. Code and trained models will be made publicly available via the NVIDIA NeMo framework

Speech foundation models have achieved state-of-the-art (SoTA) performance across various tasks, such as automatic speech recognition (ASR) in hundreds of languages. However, multi-speaker ASR remains a challenging task for these models due to data scarcity and sparsity. In this paper, we present approaches to enable speech foundation models to process and understand multi-speaker speech with limited training data. Specifically, we adapt a speech foundation model for the multi-speaker ASR task using only telephonic data. Remarkably, the adapted model also performs well on meeting data without any fine-tuning, demonstrating the generalization ability of our approach. We conduct several ablation studies to analyze the impact of different parameters and strategies on model performance. Our findings highlight the effectiveness of our methods. Results show that less parameters give better overall cpWER, which, although counter-intuitive, provides insights into adapting speech foundation models for multi-speaker ASR tasks with minimal annotated data., Comment: Accepted by SLT 2024

Self-supervised learning has been proved to benefit a wide range of speech processing tasks, such as speech recognition/translation, speaker verification and diarization, etc. However, most of current approaches are computationally expensive. In this paper, we propose a simplified and more efficient self-supervised learning framework termed as NeMo Encoder for Speech Tasks (NEST). Specifically, we adopt the FastConformer architecture with 8x sub-sampling rate, which is faster than Transformer or Conformer architectures. Instead of clustering-based quantization, we use fixed random projection for its simplicity and effectiveness. We also implement a generalized noisy speech augmentation that teaches the model to disentangle the main speaker from noise or other speakers. Experiments show that \model improves over existing self-supervised models and achieves new state-of-the-art performance on a variety of speech processing tasks, such as speech recognition/translation, speaker diarization, spoken language understanding, etc. Code and checkpoints will be publicly available via NVIDIA NeMo framework.

There is a significant demand for indoor localization technology in smart buildings, and the most promising solution in this field is using RF sensors and fingerprinting-based methods that employ machine learning models trained on crowd-sourced user data gathered from IoT devices. However, this raises security and privacy issues in practice. Some researchers propose to use federated learning to partially overcome privacy problems, but there still remain security concerns, e.g., single-point failure and malicious attacks. In this paper, we propose a framework named DFLoc to achieve precise 3D localization tasks while considering the following two security concerns. Particularly, we design a specialized blockchain to decentralize the framework by distributing the tasks such as model distribution and aggregation which are handled by a central server to all clients in most previous works, to address the issue of the single-point failure for a reliable and accurate indoor localization system. Moreover, we introduce an updated model verification mechanism within the blockchain to alleviate the concern of malicious node attacks. Experimental results substantiate the framework's capacity to deliver accurate 3D location predictions and its superior resistance to the impacts of single-point failure and malicious attacks when compared to conventional centralized federated learning systems.

The structure of proteins is the basis for studying protein function and drug design. The emergence of AlphaFold 2 has greatly promoted the prediction of protein 3D structures, and it is of great significance to give an overall and accurate evaluation of the predicted models, especially the complex models. Among the existing methods for evaluating multimer structures, DockQ is the most commonly used. However, as a more suitable metric for complex docking, DockQ cannot provide a unique and accurate evaluation in the non-docking situation. Therefore, it is necessary to propose an evaluation strategy that can directly evaluate the whole complex without limitation and achieve good results. In this work, we proposed DCI score, a new evaluation strategy for protein complex structure models, which only bases on distance map and CI (contact-interface) map, DCI focuses on the prediction accuracy of the contact interface based on the overall evaluation of complex structure, is not inferior to DockQ in the evaluation accuracy according to CAPRI classification, and is able to handle the non-docking situation better than DockQ. Besides, we calculated DCI score on CASP datasets and compared it with CASP official assessment, which obtained good results. In addition, we found that DCI can better evaluate the overall structure deviation caused by interface prediction errors in the case of multi-chains. Our DCI is available at \url{https://gitee.com/WendaWang/DCI-score.git}, and the online-server is available at \url{http://mialab.ruc.edu.cn/DCIServer/}.

Incorporating speech understanding capabilities into pretrained large-language models has become a vital research direction (SpeechLLM). The previous architectures can be categorized as: i) GPT-style, prepend speech prompts to the text prompts as a sequence of LLM inputs like a decoder-only model; ii) T5-style, introduce speech cross-attention to each layer of the pretrained LLMs. We propose BESTOW architecture to bring the BESt features from TwO Worlds into a single model that is highly efficient and has strong multitask capabilities. Moreover, there is no clear streaming solution for either style, especially considering the solution should generalize to speech multitask. We reformulate streamable SpeechLLM as a read-write policy problem and unifies the offline and streaming research with BESTOW architecture. Hence we demonstrate the first open-source SpeechLLM solution that enables Streaming and Multitask at scale (beyond ASR) at the same time. This streamable solution achieves very strong performance on a wide range of speech tasks (ASR, AST, SQA, unseen DynamicSuperb). It is end-to-end optimizable, with lower training/inference cost, and demonstrates LLM knowledge transferability to speech.

Recent advances in speech recognition and translation rely on hundreds of thousands of hours of Internet speech data. We argue that state-of-the art accuracy can be reached without relying on web-scale data. Canary - multilingual ASR and speech translation model, outperforms current state-of-the-art models - Whisper, OWSM, and Seamless-M4T on English, French, Spanish, and German languages, while being trained on an order of magnitude less data than these models. Three key factors enables such data-efficient model: (1) a FastConformer-based attention encoder-decoder architecture (2) training on synthetic data generated with machine translation and (3) advanced training techniques: data-balancing, dynamic data blending, dynamic bucketing and noise-robust fine-tuning. The model, weights, and training code will be open-sourced., Comment: Accepted at Interspeech-2024

We introduce JuliVQC: a light-weight, yet extremely efficient variational quantum circuit simulator. JuliVQC is part of an effort for classical simulation of the \textit{Zuchongzhi} quantum processors, where it is extensively used to characterize the circuit noises, as a building block in the Schr$\ddot{\text{o}}$dinger-Feynman algorithm for classical verification and performance benchmarking, and for variational optimization of the Fsim gate parameters. The design principle of JuliVQC is three-fold: (1) Transparent implementation of its core algorithms, realized by using the high-performance script language Julia; (2) Efficiency is the focus, with a cache-friendly implementation of each elementary operations and support for shared-memory parallelization; (3) Native support of automatic differentiation for both the noiseless and noisy quantum circuits. We perform extensive numerical experiments on JuliVQC in different application scenarios, including quantum circuits, variational quantum circuits and their noisy counterparts, which show that its performance is among the top of the popular alternatives., Comment: 12 pages, 8 figures

Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities, enabling SLMs to interpret and generate comprehensive natural language descriptions, thereby facilitating the capability to understand both linguistic and non-linguistic features in speech. Enhanced with the proposed approach, our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. Moreover, we discover that the aligned model exhibits a zero-shot instruction-following capability without explicit speech instruction tuning. These findings highlight the potential to reshape instruction-following SLMs by incorporating rich, descriptive speech captions., Comment: Accepted to Interspeech 2024

To make practical quantum algorithms work, large-scale quantum processors protected by error-correcting codes are required to resist noise and ensure reliable computational outcomes. However, a major challenge arises from defects in processor fabrication, as well as occasional losses or cosmic rays during the computing process, all of which can lead to qubit malfunctions and disrupt error-correcting codes' normal operations. In this context, we introduce an automatic adapter to implement the surface code on defective lattices. Unlike previous approaches, this adapter leverages newly proposed bandage-like super-stabilizers to save more qubits when defects are clustered, thus enhancing the code distance and reducing super-stabilizer weight. For instance, in comparison with earlier methods, with a code size of 27 and a random defect rate of 2\%, the disabled qubits decrease by $1/3$, and the average preserved code distance increases by 63\%. This demonstrates a significant reduction in overhead when handling defects using our approach, and this advantage amplifies with increasing processor size and defect rates. Our work presents a low-overhead, automated solution to the challenge of adapting the surface code to defects, an essential step towards scaling up the construction of large-scale quantum computers for practical applications.

In this paper, we perform the phase space analysis to investigate the evolution of the early universe in Einstein-Cartan theory. By studying the stability of critical points in dynamical system, it is found that there exist two stable critical points which represent an expanding solution and an Einstein static solution respectively. After analyzing the phase diagram of the dynamical system, we find that there may exist a bouncing universe, an oscillating universe or an Einstein static universe in the early time of universe. In addition, by assuming that the early universe filled by the radiation with $\omega= 1/3$ , the initial states of the early universe are Einstein static universe or oscillating universe. When the equation of state $\omega$ decreases with time, the universe can exit from the initial state and evolve into an expanding phase.

We implement an experiment on a photonic quantum processor establishing efficacy of the elementary quantum system in classical information storage. The advantage is established by considering a class of simple bipartite games played with the communication resource qubit and classical bit (c-bit), respectively. Conventional wisdom, supported by the no-go theorems of Holevo and Frenkel-Weiner, suggests that such a quantum advantage is unattainable when the sender and receiver share randomness or classical correlations. However, our results reveal a quantum advantage in a scenario devoid of any shared randomness. Our experiment involves the development of a variational triangular polarimeter, enabling the realization of positive operator value measurements crucial for establishing the targeted quantum advantage. Beyond showcasing a robust communication advantage with a single qubit, our work paves the way for immediate applications in near-term quantum technologies. It provides a semi-device-independent certification scheme for quantum encoding-decoding systems and offers an efficient method for information loading and transmission in quantum networks., Comment: Accepted for publication in Phys. Rev. Lett

We investigate the constant-roll inflation and the evolution of primordial black holes (PBHs) with Barrow holographic dark energy (BHDE). Using the modified Friedmann equation and the constant-roll condition in BHDE model, we calculate the constant-roll parameters, the scalar spectral index parameter and the tensor-to-scalar ratio with the chaotic potential $V_{0}\phi^{n}$. Then, we show that a suitable value of the power exponent is $n=1$ by using the Planck 2018 data. Considering the accretion process and the evaporation due to Hawking radiation, we discuss the evolution of PBHs in BHDE model and obtain that the PBHs mass is in the mass window of PBHs.

Quantum state readout serves as the cornerstone of quantum information processing, exerting profound influence on quantum communication, computation, and metrology. In this study, we introduce an innovative readout architecture called Compression Shadow (CompShadow), which transforms the conventional readout paradigm by compressing multi-qubit states into single-qubit shadows before measurement. Compared to direct measurements of the initial quantum states, CompShadow achieves comparable accuracy in amplitude and observable expectation estimation while consuming similar measurement resources. Furthermore, its implementation on near-term quantum hardware with nearest-neighbor coupling architectures is straightforward. Significantly, CompShadow brings forth novel features, including the complete suppression of correlated readout noise, fundamentally reducing the quantum hardware demands for readout. It also facilitates the exploration of multi-body system properties through single-qubit probes and opens the door to designing quantum communication protocols with exponential loss suppression. Our findings mark the emergence of a new era in quantum state readout, setting the stage for a revolutionary leap in quantum information processing capabilities.

Visible transparent but infrared reflective materials are ideal candidates for both transparent conductive films and low-emissivity glass, which are highly desired in a broad variety of areas such as touchscreens and displays, photovoltaics, smart windows, and antistatic coatings. Ultrathin Ti3C2Tx MXene films are emerging as promising low-emissivity transparent candidates. However, the fundamental IR properties of Ti3C2Tx has not been revealed experimentally due to daunting challenges in the preparation of continuous, large-area, and ultrathin films of optical quality on flat substrates. Herein, we proposed a tape-free transfer method that can help prepare centimeter-size and ultrathin (down to 8 nm) Ti3C2Tx films on diverse optical substrates. Benefitting from this method, the refractive index and permittivity for Ti3C2Tx were successfully measured. Ti3C2Tx films exhibit large in-plane permittivity in the IR region, yielding maximum IR reflectance of 88% for bulk films. Interestingly, three anomalies were found in ultrathin Ti3C2Tx films: strong dispersion in the permittivity, interlayer space-dependent optical properties, and abnormally high IR absorption for a 15-nm-thick film. These anomalies are important guidelines in the design of Ti3C2Tx-based low-emissivity transparent films and other related devices, and may inspire other intriguing applications such as ultrathin IR absorption coatings and tunable IR optical devices.