Search

The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness., Comment: arXiv admin note: text overlap with arXiv:2404.17808

The concept of 6G distributed integrated sensing and communications (DISAC) builds upon the functionality of integrated sensing and communications (ISAC) by integrating distributed architectures, significantly enhancing both sensing and communication coverage and performance. In 6G DISAC systems, tracking target trajectories requires base stations (BSs) to hand over their tracked targets to neighboring BSs. Determining what information to share, where, how, and when is critical to effective handover. This paper addresses the target handover challenge in DISAC systems and introduces a method enabling BSs to share essential target trajectory information at appropriate time steps, facilitating seamless handovers to other BSs. The target tracking problem is tackled using the standard trajectory Poisson multi-Bernoulli mixture (TPMBM) filter, enhanced with the proposed handover algorithm. Simulation results confirm the effectiveness of the implemented tracking solution., Comment: Submitted to ICC 2025

Deep neural networks (DNNs) have recently achieved impressive success across a wide range of real-world vision and language processing tasks, spanning from image classification to many other downstream vision tasks, such as object detection, tracking, and segmentation. However, previous well-established DNNs, despite being able to maintain superior accuracy, have also been evolving to be deeper and wider and thus inevitably necessitate prohibitive computational resources for both training and inference. This trend further enlarges the computational gap between computation-intensive DNNs and resource-constrained embedded computing systems, making it challenging to deploy powerful DNNs upon real-world embedded computing systems towards ubiquitous embedded intelligence. To alleviate the above computational gap and enable ubiquitous embedded intelligence, we, in this survey, focus on discussing recent efficient deep learning infrastructures for embedded computing systems, spanning from training to inference, from manual to automated, from convolutional neural networks to transformers, from transformers to vision transformers, from vision models to large language models, from software to hardware, and from algorithms to applications. Specifically, we discuss recent efficient deep learning infrastructures for embedded computing systems from the lens of (1) efficient manual network design for embedded computing systems, (2) efficient automated network design for embedded computing systems, (3) efficient network compression for embedded computing systems, (4) efficient on-device learning for embedded computing systems, (5) efficient large language models for embedded computing systems, (6) efficient deep learning software and hardware for embedded computing systems, and (7) efficient intelligent applications for embedded computing systems., Comment: ACM Transactions on Embedded Computing Systems (TECS) 2024

Marked event data captures events by recording their continuous-valued occurrence timestamps along with their corresponding discrete-valued types. They have appeared in various real-world scenarios such as social media, financial transactions, and healthcare records, and have been effectively modeled through Marked Temporal Point Process (MTPP) models. Recently, developing generative models for these MTPP models have seen rapid development due to their powerful generative capability and less restrictive functional forms. However, existing generative MTPP models are usually challenged in jointly modeling events' timestamps and types since: (1) mainstream methods design the generative mechanisms for timestamps only and do not include event types; (2) the complex interdependence between the timestamps and event types are overlooked. In this paper, we propose a novel generative MTPP model called BMTPP. Unlike existing generative MTPP models, BMTPP flexibly models marked temporal joint distributions using a parameter-based approach. Additionally, by adding joint noise to the marked temporal data space, BMTPP effectively captures and explicitly reveals the interdependence between timestamps and event types. Extensive experiments validate the superiority of our approach over other state-of-the-art models and its ability to effectively capture marked-temporal interdependence.

Isolated Sign Language Recognition (ISLR) focuses on identifying individual sign language glosses. Considering the diversity of sign languages across geographical regions, developing region-specific ISLR datasets is crucial for supporting communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale word-level dataset for the ISLR task. To fill this gap, we curate \underline{\textbf{the first}} large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan. Compared to other publicly available datasets, MM-WLAuslan exhibits three significant advantages: (1) the largest amount of data, (2) the most extensive vocabulary, and (3) the most diverse of multi-modal camera views. Specifically, we record 282K+ sign videos covering 3,215 commonly used Auslan glosses presented by 73 signers in a studio environment. Moreover, our filming system includes two different types of cameras, i.e., three Kinect-V2 cameras and a RealSense camera. We position cameras hemispherically around the front half of the model and simultaneously record videos using all four cameras. Furthermore, we benchmark results with state-of-the-art methods for various multi-modal ISLR settings on MM-WLAuslan, including multi-view, cross-camera, and cross-view. Experiment results indicate that MM-WLAuslan is a challenging ISLR dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide. All datasets and benchmarks are available at MM-WLAuslan.

Large language models (LLM) have been attracting much attention from the community recently, due to their remarkable performance in all kinds of downstream tasks. According to the well-known scaling law, scaling up a dense LLM enhances its capabilities, but also significantly increases the computational complexity. Mixture-of-Experts (MoE) models address that by allowing the model size to grow without substantially raising training or inference costs. Yet MoE models face challenges regarding knowledge sharing among experts, making their performance somehow sensitive to routing accuracy. To tackle that, previous works introduced shared experts and combined their outputs with those of the top $K$ routed experts in an ``addition'' manner. In this paper, inspired by collective matrix factorization to learn shared knowledge among data, we propose CartesianMoE, which implements more effective knowledge sharing among experts in more like a ``multiplication'' manner. Extensive experimental results indicate that CartesianMoE outperforms previous MoE models for building LLMs, in terms of both perplexity and downstream task performance. And we also find that CartesianMoE achieves better expert routing robustness.

We investigate the Ising model confined to a spherical surface, focusing on its implementation using a Fibonacci lattice. The challenge lies in uniformly covering the spherical surface to enable reliable comparisons with planar models. Monte Carlo simulations and graph convolutional networks(GCNs) are employed to analyze spin configurations at varying temperatures and to identify phase transition temperatures. Although the spherical Fibonacci lattice is sufficiently uniform, there are still some irregular sites, which introduce interesting effects. In the ferromagnetic case, sites with fewer neighbors are more likely to undergo spin flips at low temperatures; however, this is not necessarily true at high temperatures, which could explain why the phase transition temperature is higher compared to the planar Ising model. In the antiferromagnetic case, the presence of irregular sites results in the total energy of the system at zero temperature not being the lowest. Phase transition temperatures are estimated using specific heat analysis and GCNs, revealing $T_C$ values for both ferromagnetic and antiferromagnetic cases. The study underscores the significance of the Fibonacci lattice's geometric properties in understanding spin interactions in microgravity environments., Comment: 7 pages, 10 figures

Fast radio bursts (FRBs) are transient radio bursts of extragalactic origin characterized by millisecond durations and high luminosities. We report on observations of FRB 20240114A conducted with the Robert C. Byrd Green Bank Telescope (GBT) at frequencies ranging from 720 to 920 MHz. A total of 429 bursts were detected, with a single observation recording 359 bursts over 1.38 hours, corresponding to a burst rate of 260 bursts per hour. The average rotation measures (RMs) were $349.2 \pm 1.0$ rad m$^{-2}$ on February 23, 2024, and $360.4 \pm 0.4$ rad m$^{-2}$ on March 1, 2024. Of the 297 bursts with detected RMs, 72% have a linear polarization fraction greater than 90%, and 14% exhibit circular polarization with a signal-to-noise ratio $> 5$. Our sample also displayed polarization angle swings. We compare the linear polarization of FRB 20240114A with that of FRB 20201124A, FRB 20220912A, and non-repeating FRBs. The mean linear polarization fraction for non-repeating FRBs is 58%. In contrast, the mean linear polarization fraction for the three repeating FRBs is 94%, which is significantly higher than that of the non-repeating FRBs. Under the T-test, the three repeating FRBs have similar linear polarization distributions, but these distributions differ from those of the non-repeating FRBs. This suggests that non-repeating FRBs may have different emission mechanisms or are subject to depolarization., Comment: 42 pages, 5 figures. arXiv admin note: text overlap with arXiv:2304.14671

The aim of this article is to study the Auslander algebra of any representation-finite string algebra. More precisely, we introduce the notion of gluing algebras and show that the Auslander algebra of a representation-finite string algebra is a quotient of a \gluing algebra of $\vec{A}^e_n $. As applications, the Auslander algebras of two classes of string algebras whose quivers are Dynkin types $A$ and $D$ are described. Moreover, the representation types of the above Auslander algebras are also given exactly.

Temporal point processes (TPPs) are effective for modeling event occurrences over time, but they struggle with sparse and uncertain events in federated systems, where privacy is a major concern. To address this, we propose \textit{FedPP}, a Federated neural nonparametric Point Process model. FedPP integrates neural embeddings into Sigmoidal Gaussian Cox Processes (SGCPs) on the client side, which is a flexible and expressive class of TPPs, allowing it to generate highly flexible intensity functions that capture client-specific event dynamics and uncertainties while efficiently summarizing historical records. For global aggregation, FedPP introduces a divergence-based mechanism that communicates the distributions of SGCPs' kernel hyperparameters between the server and clients, while keeping client-specific parameters local to ensure privacy and personalization. FedPP effectively captures event uncertainty and sparsity, and extensive experiments demonstrate its superior performance in federated settings, particularly with KL divergence and Wasserstein distance-based global aggregation.

The detection of fast radio bursts (FRBs) in radio astronomy is a complex task due to the challenges posed by radio frequency interference (RFI) and signal dispersion in the interstellar medium. Traditional search algorithms are often inefficient, time-consuming, and generate a high number of false positives. In this paper, we present DRAFTS, a deep learning-based radio fast transient search pipeline. DRAFTS integrates object detection and binary classification techniques to accurately identify FRBs in radio data. We developed a large, real-world dataset of FRBs for training deep learning models. The search test on FAST real observation data demonstrates that DRAFTS performs exceptionally in terms of accuracy, completeness, and search speed. In the re-search of FRB 20190520B observation data, DRAFTS detected more than three times the number of bursts compared to Heimdall, highlighting the potential for future FRB detection and analysis., Comment: 20 pages, 10 figures, submitted

Reconfigurable intelligent surfaces (RISs) are key enablers for integrated sensing and communication (ISAC) systems in the 6G communication era. With the capability of dynamically shaping the channel, RISs can enhance communication coverage. Additionally, RISs can serve as additional anchors with high angular resolution to improve localization and sensing services in extreme scenarios. However, knowledge of anchors' states such as position, orientation, and hardware impairments are crucial for localization and sensing applications, requiring dedicated calibration, including geometry and hardware calibration. This paper provides an overview of various types of RIS calibration, their impacts, and the challenges they pose in ISAC systems.

Over 700 bright millisecond-duration radio transients, known as Fast Radio Bursts (FRBs), have been identified to date. Nevertheless, the origin of FRBs remains unknown. The two repeating FRBs (FRB 20121102A and FRB 20190520B) have been verified to be associated with persistent radio sources (PRSs), making them the best candidates to study the nature of FRBs. Monitoring the variability in PRSs is essential for understanding their physical nature. We conducted 22 observations of the PRSs linked to FRB 20121102A and FRB 20190520B using the Karl G. Jansky Very Large Array (VLA), to study their variability. We have observed significant flux variability for the PRSs of FRB 20121102A and FRB 20190520B, with a confidence level exceeding 99.99%, based on the observations covering the longest timescale recorded to date. The observed variability of the two PRSs exhibits no significant difference in amplitude across both short and long timescales. We found that the radio-derived star formation rates of the two FRB hosts are significantly higher than those measured by the optical $H_{\alpha}$ emissions, indicating that their host galaxies are highly obscured or most radio emissions are not from star formation processes. The observed timescale of PRS flux evolution constrained the magnetic field of FRB 20121102A with $B_\parallel\gtrsim1~{\rm mG}$ and FRB 20190520B with $B_\parallel\gtrsim0.1~{\rm mG}$., Comment: 16 pages, 7 figures, accepted by ApJ

Unsupervised anomaly detection (AD) aims to train robust detection models using only normal samples, while can generalize well to unseen anomalies. Recent research focuses on a unified unsupervised AD setting in which only one model is trained for all classes, i.e., n-class-one-model paradigm. Feature-reconstruction-based methods achieve state-of-the-art performance in this scenario. However, existing methods often suffer from a lack of sufficient contextual awareness, thereby compromising the quality of the reconstruction. To address this issue, we introduce a novel Reconstruction as Sequence (RAS) method, which enhances the contextual correspondence during feature reconstruction from a sequence modeling perspective. In particular, based on the transformer technique, we integrate a specialized RASFormer block into RAS. This block enables the capture of spatial relationships among different image regions and enhances sequential dependencies throughout the reconstruction process. By incorporating the RASFormer block, our RAS method achieves superior contextual awareness capabilities, leading to remarkable performance. Experimental results show that our RAS significantly outperforms competing methods, well demonstrating the effectiveness and superiority of our method. Our code is available at https://github.com/Nothingtolose9979/RAS.

This letter reports the first complete observation of magnetized collisionless shock precursors formed through the compression of Biermann-battery magnetic fields in laser produced plasmas. At OMEGA, lasers produce a supersonic CH plasma flow which is magnetized with Biermann-battery magnetic fields. The plasma flow collides with an unmagnetized hydrogen gas jet plasma to create a magnetized shock precursor. The situation where the flowing plasma carries the magnetic field is similar to the Venusian bow shock. Imaging 2$\omega$ Thomson scattering confirms that the interaction is collisionless and shows density and temperature jumps. Proton radiographs have regions of strong deflections and FLASH magnetohydrodynamic (MHD) simulations show the presence of Biermann fields in the Thomson scattering region. Electrons are accelerated to energies of up to 100 keV in a power-law spectrum. OSIRIS particle-in-cell (PIC) simulations, initialized with measured parameters, show the formation of a magnetized shock precursor and corroborate the experimental observables., Comment: 6 pages, 5 figures

Tooth point cloud segmentation is a fundamental task in many orthodontic applications. Current research mainly focuses on fully supervised learning which demands expensive and tedious manual point-wise annotation. Although recent weakly-supervised alternatives are proposed to use weak labels for 3D segmentation and achieve promising results, they tend to fail when the labels are extremely sparse. Inspired by the powerful promptable segmentation capability of the Segment Anything Model (SAM), we propose a framework named SAMTooth that leverages such capacity to complement the extremely sparse supervision. To automatically generate appropriate point prompts for SAM, we propose a novel Confidence-aware Prompt Generation strategy, where coarse category predictions are aggregated with confidence-aware filtering. Furthermore, to fully exploit the structural and shape clues in SAM's outputs for assisting the 3D feature learning, we advance a Mask-guided Representation Learning that re-projects the generated tooth masks of SAM into 3D space and constrains these points of different teeth to possess distinguished representations. To demonstrate the effectiveness of the framework, we conduct experiments on the public dataset and surprisingly find with only 0.1\% annotations (one point per tooth), our method can surpass recent weakly supervised methods by a large margin, and the performance is even comparable to the recent fully-supervised methods, showcasing the significant potential of applying SAM to 3D perception tasks with sparse labels. Code is available at https://github.com/CUHK-AIM-Group/SAMTooth., Comment: To appear at MICCAI24

Localization plays an increasingly pivotal role in 5G/6G systems, enabling various applications. This paper focuses on the privacy concerns associated with delay-based localization, where unauthorized base stations attempt to infer the location of the end user. We propose a method to disrupt localization at unauthorized nodes by injecting artificial components into the pilot signal, exploiting model mismatches inherent in these nodes. Specifically, we investigate the effectiveness of two techniques, namely artificial multipath (AM) and artificial noise (AN), in mitigating location leakage. By leveraging the misspecified Cram\'er-Rao bound framework, we evaluate the impact of these techniques on unauthorized localization performance. Our results demonstrate that pilot manipulation significantly degrades the accuracy of unauthorized localization while minimally affecting legitimate localization. Moreover, we find that the superiority of AM over AN varies depending on the specific scenario., Comment: 6pages, conference paper

Recent advancements in autonomous driving, augmented reality, robotics, and embodied intelligence have necessitated 3D perception algorithms. However, current 3D perception methods, particularly small models, struggle with processing logical reasoning, question-answering, and handling open scenario categories. On the other hand, generative multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks, due to weak spatial and local object perception, poor text-based geometric numerical output, and inability to handle camera focal variations. To address these challenges, we propose the following solutions: Spatial-Enhanced Local Feature Mining for better spatial feature extraction, 3D Query Token-Derived Info Decoding for precise geometric regression, and Geometry Projection-Based 3D Reasoning for handling camera focal length variations. We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM. Additionally, we have constructed the IG3D dataset, which provides fine-grained descriptions and question-answer annotations. Extensive experiments demonstrate that our LLMI3D achieves state-of-the-art performance, significantly outperforming existing methods.

Central to the layered kagome lattice superconductors AV3Sb5 (A = K, Cs, Rb) is a cascade of novel quantum states triggered by an unconventional charge density wave (CDW) order. The three-dimensional (3D) order involves a 2x2x2 phase coherent stacking of 2x2 charge density modulations in the kagome plane at low temperatures, exhibiting a CDW energy gap and evidence for time-reversal symmetry breaking. Here we report the discovery of emergent superconductivity and primary pair density wave (PDW) at the antiphase boundaries and stacking faults of bulk CDW order. We find that the {\pi}-phase shift dislocations can naturally appear on the surface as the Cs atoms form 2x2 superstructures that are out of phase with the bulk CDW. An incipient narrow band of surface states inside bulk CDW gap emerge close to the Fermi level where a particle-hole symmetric energy gap develops. We demonstrate that the energy gap originates from a novel quasi-2D kagome superconducting state (Tc ~ 5.4 K) intertwined with bulk CDW order, exhibiting an unprecedented vortex core spectrum and spatial modulations of the superconducting gap consistent with a 4x4 PDW. Intriguingly, the 2D kagome superconductivity is shown to be tunable on and off by atomically manipulating the Cs atoms on the surface. Our findings provide fresh new insights for understanding the interplay between the unconventional CDW and superconductivity in kagome metals and a pathway for atomic manipulation and topological defects engineering of quantum many-body states in correlated materials.

Understanding and engineering the coupling of spins in nanomaterials is of central importance for designing novel devices. Graphene nanostructures with {\pi}-magnetism offer a chemically tunable platform to explore quantum magnetic interactions. However, realizing spin chains bearing controlled odd-even effects with suitable nanographene systems is challenging. Here, we demonstrate the successful on-surface synthesis of spin-1/2 antiferromagnetic Heisenberg chains with parity-dependent magnetization based on antiaromatic diaza-hexa-peri-hexabenzocoronene (diaza-HBC) units. Using distinct synthetic strategies, two types of spin chains with different terminals were synthesized, both exhibiting a robust odd-even effect on the spin coupling along the chain. Combined investigations using scanning tunneling microscopy, non-contact atomic force microscopy, density functional theory calculations, and quantum spin models confirmed the structures of the diaza-HBC chains and revealed their magnetic properties, which has an S = 1/2 spin per unit through electron donation from the diaza-HBC core to the Au(111) substrate. Gapped excitations were observed in even-numbered chains, while enhanced Kondo resonance emerged in odd-numbered units of odd-numbered chains due to the redistribution of the unpaired spin along the chain. Our findings provide an effective strategy to construct nanographene spin chains and unveil the odd-even effect in their magnetic properties, offering potential applications in nanoscale spintronics.

Domain Adaptation (DA) facilitates knowledge transfer from a source domain to a related target domain. This paper investigates a practical DA paradigm, namely Source data-Free Active Domain Adaptation (SFADA), where source data becomes inaccessible during adaptation, and a minimum amount of annotation budget is available in the target domain. Without referencing the source data, new challenges emerge in identifying the most informative target samples for labeling, establishing cross-domain alignment during adaptation, and ensuring continuous performance improvements through the iterative query-and-adaptation process. In response, we present learn from the learnt (LFTL), a novel paradigm for SFADA to leverage the learnt knowledge from the source pretrained model and actively iterated models without extra overhead. We propose Contrastive Active Sampling to learn from the hypotheses of the preceding model, thereby querying target samples that are both informative to the current model and persistently challenging throughout active learning. During adaptation, we learn from features of actively selected anchors obtained from previous intermediate models, so that the Visual Persistence-guided Adaptation can facilitate feature distribution alignment and active sample exploitation. Extensive experiments on three widely-used benchmarks show that our LFTL achieves state-of-the-art performance, superior computational efficiency and continuous improvements as the annotation budget increases. Our code is available at https://github.com/lyumengyao/lftl., Comment: ECCV 2024

Compared to classical optical coherence theory based on Maxwell's electromagnetic theory and Glauber's quantum optical coherence theory based on matrix mechanics formulation of quantum mechanics, quantum optical coherence theory based on Feynman's path integral formulation of quantum mechanics provides a novel tool to study optical coherence. It has the advantage of understanding the connection between mathematical calculations and physical interpretations better. Quantum optical coherence theory based on Feynman's path integral is introduced and reviewed in this paper. Based on the results of transient first-order interference of two independent light beams, it is predicted that the classical model for electric field of thermal light introduced by classical optical textbooks may not be accurate. The physics of two-photon bunching of thermal light and Hong-Ou-Mandel dip of entangled photon pairs is the same, which can be interpreted by constructive and destructive two-photon interference, respectively. Quantum optical coherence theory based on Feynman's path integral is helpful to understand the coherence properties of light, which may eventually lead us to the answer of the question: what is a photon?, Comment: 40 pages, 35 figures

In smart city development, the automatic detection of structures and vehicles within urban or suburban areas via array radar (airborne or vehicle platforms) becomes crucial. However, the inescapable multipath effect adversely affects the radar's capability to detect and track targets. Frequency Diversity Array (FDA)-MIMO radar offers innovative solutions in mitigating multipath due to its frequency flexibility and waveform diversity traits amongst array elements. Hence, utilizing FDA-MIMO radar, this research proposes a multipath discrimination and suppression strategy to augment target detection and suppress false alarms. The primary advancement is the transformation of conventional multipath suppression into a multipath recognition issue, thereby enabling multipath components from single-frame echo data to be separated without prior knowledge. By offsetting the distance steering vectors of different objects to be detected, the accurate spectral information corresponding to the current distance unit can be extracted during spatial spectrum estimation. The direct and multipath components are differentiated depending on whether the transmitting and receiving angles match. Additionally, to mitigate high-order multipath, the echo intensity of multipath components is reduced via joint optimization of array transmit weighting and frequency increment. The numerical results show that the proposed algorithm can identify multipath at different distances in both single-target and multi-target scenarios, which is superior to the general MIMO radar., Comment: 14 pages

In the past few years, large-scale pre-trained vision-language models like CLIP have achieved tremendous success in various fields. Naturally, how to transfer the rich knowledge in such huge pre-trained models to downstream tasks and datasets becomes a hot topic. During downstream adaptation, the most challenging problems are overfitting and catastrophic forgetting, which can cause the model to overly focus on the current data and lose more crucial domain-general knowledge. Existing works use classic regularization techniques to solve the problems. As solutions become increasingly complex, the ever-growing storage and inference costs are also a significant problem that urgently needs to be addressed. While in this paper, we start from an observation that proper random noise can suppress overfitting and catastrophic forgetting. Then we regard quantization error as a kind of noise, and explore quantization for regularizing vision-language model, which is quite efficiency and effective. Furthermore, to improve the model's generalization capability while maintaining its specialization capacity at minimal cost, we deeply analyze the characteristics of the weight distribution in prompts, conclude several principles for quantization module design and follow such principles to create several competitive baselines. The proposed method is significantly efficient due to its inherent lightweight nature, making it possible to adapt on extremely resource-limited devices. Our method can be fruitfully integrated into many existing approaches like MaPLe, enhancing accuracy while reducing storage overhead, making it more powerful yet versatile. Extensive experiments on 11 datasets shows great superiority of our method sufficiently. Code is available at https://github.com/beyondhtx/QPrompt., Comment: ECCV 2024

Scaling the size of a model enhances its capabilities but significantly increases computation complexity. Mixture-of-Experts models (MoE) address the issue by allowing model size to scale up without substantially increasing training or inference costs. In MoE, there is an important module called the router, which is used to distribute each token to the experts. Currently, the mainstream routing methods include dynamic routing and fixed routing. Despite their promising results, MoE models encounter several challenges. Primarily, for dynamic routing methods, the dispersion of training tokens across multiple experts can lead to underfitting, particularly for infrequent tokens. Additionally, though fixed routing methods can mitigate that issue, they compromise on the diversity of representations. In this paper, we propose \textbf{MaskMoE}, a method designed to enhance token-level learning by employing a routing \textbf{mask}ing technique within the \textbf{M}ixture-\textbf{o}f-\textbf{E}xperts model. MaskMoE is capable of maintaining representation diversity while achieving more comprehensive training. Experimental results demonstrate that our method outperforms previous dominant Mixture-of-Experts models in terms of both perplexity (PPL) and downstream task performance., Comment: Work in progress

We discuss the use of angle of arrival (AoA) as an authentication measure in analog array multiple-input multiple-output (MIMO) systems. A base station equipped with an analog array authenticates users based on the AoA estimated from certified pilot transmissions, while active attackers manipulate their transmitted signals to mount impersonation attacks. We study several attacks of increasing intensity (captured through the availability of side information at the attackers) and assess the performance of AoA-based authentication using one-class classifiers. Our results show that some attack techniques with knowledge of the combiners at the verifier are effective in falsifying the AoA and compromising the security of the considered type of physical layer authentication., Comment: 25th IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC 2024)

Topological superconductivity (TSC) is a promising platform to host Majorana zero mode (MZM) for topological quantum computing. Recently, the noble metal alloy AuSn$_{4}$ has been identified as an intrinsic surface TSC. However, the atomic visualization of its nontrivial surface states and MZM remains elusive. Here, we report the direct observation of unconventional surface states and vortex zero mode at the gold (Au) terminated surfaces of AuSn$_{4}$, by ultra-low scanning tunneling microscope/spectroscopy. Distinct from the trivial metallic bulk states at tin (Sn) surfaces, the Au terminated surface exhibits pronounced surface states near Fermi level. Our density functional theory calculations indicate that these states arise from unconventional Rashba bands, where two Fermi circles from different bands share identical helical spin textures, chiralities, and group velocities in the same direction. Furthermore, we find that although the superconducting gap, critical temperature, anisotropic in-plane critical field are almost identical on Au and Sn terminated surfaces, the in-gap bound states inside Abrikosov vortex cores show significant differences. The vortex on Sn terminated surfaces exhibits a conventional Caroli-de Gennes-Matricon bound state while the Au surface shows a sharp zero-energy core state with a long non-splitting distance, resembling an MZM in a non-quantum-limit condition. This distinction may result from the dominant contribution of unconventional Rashba bands near Fermi energy from Au terminated surface. Our results provide a new platform for studying unconventional Rashba band and MZM in superconductors., Comment: 17 pages, 4 figures

The digital correlator is one of the most crucial data processing components of a radio telescope array. With the scale of radio interferometeric array growing, many efforts have been devoted to developing a cost-effective and scalable correlator in the field of radio astronomy. In this paper, a 192-input digital correlator with six CASPER ROACH2 boards and seven GPU servers has been deployed as the digital signal processing system for Tianlai cylinder pathfinder located in Hongliuxia observatory. The correlator consists of 192 input signals (96 dual-polarization), 125-MHz bandwidth, and full-Stokes output. The correlator inherits the advantages of the CASPER system, for example, low cost, high performance, modular scalability, and a heterogeneous computing architecture. With a rapidly deployable ROACH2 digital sampling system, a commercially expandable 10 Gigabit switching network system, and a flexible upgradable GPU computing system, the correlator forms a low-cost and easily-upgradable system, poised to support scalable large-scale interferometeric array in the future., Comment: 12 pages, 13 figures, accepted for publication in Frontiers in Astronomy and Space Sciences, section Astronomical Instrumentation

Sidelink positioning research predominantly focuses on the snapshot positioning problem, often within the mmWave band. Only a limited number of studies have delved into vehicle-to-anything (V2X) tracking within sub-6 GHz bands. In this paper, we investigate the V2X sidelink tracking challenges over sub-6 GHz frequencies. We propose a Kalman-filter-based tracking approach that leverages the estimated error covariance lower bounds (EECLBs) as measurement covariance, alongside a gating method to augment tracking performance. Through simulations employing ray-tracing data and super-resolution channel parameter estimation, we validate the feasibility of sidelink tracking using our proposed tracking filter with two novel EECLBs. Additionally, we demonstrate the efficacy of the gating method in identifying line-of-sight paths and enhancing tracking performance.

We derive the explicit Poisson kernel of Stokes equations in the half space with nonhomogeneous Navier boundary condition (BC) for both infinite and finite slip length. By using this kernel, for any $q>1$, we construct a finite energy solution of Stokes equations with Navier BC in the half space, with bounded velocity and velocity gradient, but having unbounded second derivatives in $L^q$ locally near the boundary. While the Caccioppoli type inequality of Stokes equations with Navier BC is true for the first derivatives of velocity, which is proved by us in [CPAA 2023], this example shows that the corresponding inequality for the second derivatives of the velocity is not true. Moreover, we give an alternative proof of the blow-up using a shear flow example, which is simple and is the solution of both Stokes and Navier--Stokes equations.

Disordered speech recognition profound implications for improving the quality of life for individuals afflicted with, for example, dysarthria. Dysarthric speech recognition encounters challenges including limited data, substantial dissimilarities between dysarthric and non-dysarthric speakers, and significant speaker variations stemming from the disorder. This paper introduces Perceiver-Prompt, a method for speaker adaptation that utilizes P-Tuning on the Whisper large-scale model. We first fine-tune Whisper using LoRA and then integrate a trainable Perceiver to generate fixed-length speaker prompts from variable-length inputs, to improve model recognition of Chinese dysarthric speech. Experimental results from our Chinese dysarthric speech dataset demonstrate consistent improvements in recognition performance with Perceiver-Prompt. Relative reduction up to 13.04% in CER is obtained over the fine-tuned Whisper., Comment: Accepted by interspeech 2024

Remote heart rate measurement is an increasingly concerned research field, usually using remote photoplethysmography (rPPG) to collect heart rate information through video data collection. However, in certain specific scenarios (such as low light conditions, intense lighting, and non-line-of-sight situations), traditional imaging methods fail to capture image information effectively, that may lead to difficulty or inability in measuring heart rate. To address these limitations, this study proposes using ghost imaging as a substitute for traditional imaging in the aforementioned scenarios. The mean absolute error between experimental measurements and reference true values is 4.24 bpm.Additionally, the bucket signals obtained by the ghost imaging system can be directly processed using digital signal processing techniques, thereby enhancing personal privacy protection., Comment: 4 pages, 6 figures

The recently proposed Bayesian Flow Networks~(BFNs) show great potential in modeling parameter spaces, offering a unified strategy for handling continuous, discretized, and discrete data. However, BFNs cannot learn high-level semantic representation from the parameter space since {common encoders, which encode data into one static representation, cannot capture semantic changes in parameters.} This motivates a new direction: learning semantic representations hidden in the parameter spaces to characterize mixed-typed noisy data. {Accordingly, we propose a representation learning framework named ParamReL, which operates in the parameter space to obtain parameter-wise latent semantics that exhibit progressive structures. Specifically, ParamReL proposes a \emph{self-}encoder to learn latent semantics directly from parameters, rather than from observations. The encoder is then integrated into BFNs, enabling representation learning with various formats of observations. Mutual information terms further promote the disentanglement of latent semantics and capture meaningful semantics simultaneously.} We illustrate {conditional generation and reconstruction} in ParamReL via expanding BFNs, and extensive {quantitative} experimental results demonstrate the {superior effectiveness} of ParamReL in learning parameter representation.

Energy dispersive X-ray (EDX) spectroscopy in the transmission electron microscope is a key tool for nanomaterials analysis, providing a direct link between spatial and chemical information. However, using it for precisely determining chemical compositions presents challenges of noisy data from low X-ray yields and mixed signals from phases that overlap along the electron beam trajectory. Here, we introduce a novel method, non-negative matrix factorisation based pan-sharpening (PSNMF), to address these limitations. Leveraging the Poisson nature of EDX spectral noise and binning operations, PSNMF retrieves high quality phase spectral and spatial signatures via consecutive factorisations. After validating PSNMF with synthetic datasets of different noise levels, we illustrate its effectiveness on two distinct experimental cases: a nano-mineralogical lamella, and supported catalytic nanoparticles. Not only does PSNMF obtain accurate phase signatures, datasets reconstructed from the outputs have demonstrably lower noise and better fidelity than from the benchmark denoising method of principle component analysis.

Over the past years, YOLOs have emerged as the predominant paradigm in the field of real-time object detection owing to their effective balance between computational cost and detection performance. Researchers have explored the architectural designs, optimization objectives, data augmentation strategies, and others for YOLOs, achieving notable progress. However, the reliance on the non-maximum suppression (NMS) for post-processing hampers the end-to-end deployment of YOLOs and adversely impacts the inference latency. Besides, the design of various components in YOLOs lacks the comprehensive and thorough inspection, resulting in noticeable computational redundancy and limiting the model's capability. It renders the suboptimal efficiency, along with considerable potential for performance improvements. In this work, we aim to further advance the performance-efficiency boundary of YOLOs from both the post-processing and model architecture. To this end, we first present the consistent dual assignments for NMS-free training of YOLOs, which brings competitive performance and low inference latency simultaneously. Moreover, we introduce the holistic efficiency-accuracy driven model design strategy for YOLOs. We comprehensively optimize various components of YOLOs from both efficiency and accuracy perspectives, which greatly reduces the computational overhead and enhances the capability. The outcome of our effort is a new generation of YOLO series for real-time end-to-end object detection, dubbed YOLOv10. Extensive experiments show that YOLOv10 achieves state-of-the-art performance and efficiency across various model scales. For example, our YOLOv10-S is 1.8$\times$ faster than RT-DETR-R18 under the similar AP on COCO, meanwhile enjoying 2.8$\times$ smaller number of parameters and FLOPs. Compared with YOLOv9-C, YOLOv10-B has 46\% less latency and 25\% fewer parameters for the same performance., Comment: Code: https://github.com/THU-MIG/yolov10; NeurIPS 2024 Camera-ready Version

The beamforming technology with large holographic antenna arrays is one of the key enablers for the next generation of wireless systems, which can significantly improve the spectral efficiency. However, the deployment of large antenna arrays implies high algorithm complexity and resource overhead at both receiver and transmitter ends. To address this issue, advanced technologies such as artificial intelligence have been developed to reduce beamforming overhead. Intuitively, if we can implement the near-optimal beamforming only using a tiny subset of the all channel information, the overhead for channel estimation and beamforming would be reduced significantly compared with the traditional beamforming methods that usually need full channel information and the inversion of large dimensional matrix. In light of this idea, we propose a novel scheme that utilizes Wasserstein generative adversarial network with gradient penalty to infer the full beamforming matrices based on very little of channel information. Simulation results confirm that it can accomplish comparable performance with the weighted minimum mean-square error algorithm, while reducing the overhead by over 50%.

In many practical applications, it is often difficult and expensive to obtain large-scale labeled data to train state-of-the-art deep neural networks. Therefore, transferring the learned knowledge from a separate, labeled source domain to an unlabeled or sparsely labeled target domain becomes an appealing alternative. However, direct transfer often results in significant performance decay due to domain shift. Domain adaptation (DA) aims to address this problem by aligning the distributions between the source and target domains. Multi-source domain adaptation (MDA) is a powerful and practical extension in which the labeled data may be collected from multiple sources with different distributions. In this survey, we first define various MDA strategies. Then we systematically summarize and compare modern MDA methods in the deep learning era from different perspectives, followed by commonly used datasets and a brief benchmark. Finally, we discuss future research directions for MDA that are worth investigating., Comment: Accepted by IJCAI 2024. arXiv admin note: text overlap with arXiv:2002.12169

Based on the Maxwell-Beatty reciprocity theorem, static non-reciprocity has been realized by using nonlinearity, but this non-reciprocity has strict restrictions on input amplitude and structure size (number of units). Here, we propose a robotic metamaterial with two components of displacement and rotation, which uses active control to add external forces on the units to break reciprocity at the level of the interactions between the units. We show analytically and simulatively that breaking reciprocity at the level of the interactions directly leads to a strong asymmetric response of displacement in a static system, this displacement-specific characteristic not only has no restrictions on size, input amplitude, and suitable geometric asymmetry, but also can be transmitted to rotation by coupling under large deformation. After the evolution from statics to dynamics, asymmetric transmission and unidirectional amplification of vector solitons are both implemented in this system. Our research uncovers the evolution of static non-reciprocity to dynamic non-reciprocity while building a bridge between non-reciprocity physics and soliton science.

Byte Pair Encoding (BPE) serves as a foundation method for text tokenization in the Natural Language Processing (NLP) field. Despite its wide adoption, the original BPE algorithm harbors an inherent flaw: it inadvertently introduces a frequency imbalance for tokens in the text corpus. Since BPE iteratively merges the most frequent token pair in the text corpus to generate a new token and keeps all generated tokens in the vocabulary, it unavoidably holds tokens that primarily act as components of a longer token and appear infrequently on their own. We term such tokens as Scaffold Tokens. Due to their infrequent occurrences in the text corpus, Scaffold Tokens pose a learning imbalance issue. To address that issue, we propose Scaffold-BPE, which incorporates a dynamic scaffold token removal mechanism by parameter-free, computation-light, and easy-to-implement modifications to the original BPE method. This novel approach ensures the exclusion of low-frequency Scaffold Tokens from the token representations for given texts, thereby mitigating the issue of frequency imbalance and facilitating model training. On extensive experiments across language modeling and even machine translation, Scaffold-BPE consistently outperforms the original BPE, well demonstrating its effectiveness.

Recently, Large Language Models (LLMs) have been widely adopted in a wide range of tasks, leading to increasing attention towards the research on how scaling LLMs affects their performance. Existing works, termed Scaling Laws, have discovered that the final test loss of LLMs scales as power-laws with model size, computational budget, and dataset size. However, the temporal change of the test loss of an LLM throughout its pre-training process remains unexplored, though it is valuable in many aspects, such as selecting better hyperparameters \textit{directly} on the target LLM. In this paper, we propose the novel concept of Temporal Scaling Law, studying how the test loss of an LLM evolves as the training steps scale up. In contrast to modeling the test loss as a whole in a coarse-grained manner, we break it down and dive into the fine-grained test loss of each token position, and further develop a dynamic hyperbolic-law. Afterwards, we derive the much more precise temporal scaling law by studying the temporal patterns of the parameters in the dynamic hyperbolic-law. Results on both in-distribution (ID) and out-of-distribution (OOD) validation datasets demonstrate that our temporal scaling law accurately predicts the test loss of LLMs across training steps. Our temporal scaling law has broad practical applications. First, it enables direct and efficient hyperparameter selection on the target LLM, such as data mixture proportions. Secondly, viewing the LLM pre-training dynamics from the token position granularity provides some insights to enhance the understanding of LLM pre-training., Comment: 8 pages, 3 figures; Under review

We present the development of a new algorithm which combines state-of-the-art energy-dispersive X-ray (EDX) spectroscopy theory and a suitable machine learning formulation for the hyperspectral unmixing of scanning transmission electron microscope EDX spectrum images. The algorithm is based on non-negative matrix factorization (NMF) incorporating a physics-guided factorization model. It optimizes a Poisson likelihood, under additional simplex constraint together with user-chosen sparsity-inducing and smoothing regularizations, and is based on iterative multiplicative updates. The fluorescence of X-rays is fully modeled thanks to state-of-the-art theoretical work. It is shown that the output of the algorithm can be used for a direct chemical quantification. With this approach, it is straightforward to include a priori knowledge on the specimen such as the presence or absence of certain chemical elements in some of its phases. This work is implemented within two open-source Python packages, espm and emtables, which are used here for data simulation, data analysis and quantification. Using simulated data, we demonstrate that incorporating physical modeling in the decomposition helps retrieve meaningful components from spatially and spectrally mixed phases, even when the data are very noisy. For synthetic data with a higher signal, the regularizations yield a tenfold increase in the quality of the reconstructed abundance maps compared to standard NMF. Our approach is further validated on experimental data with a known ground truth, where state-of-the art results are achieved by using prior knowledge about the sample. Our model can be generalized to any other scanning spectroscopy techniques where underlying physical modeling can be linearized., Comment: 30 pages, 4 figures

While deep neural networks (DNNs) based personalized federated learning (PFL) is demanding for addressing data heterogeneity and shows promising performance, existing methods for federated learning (FL) suffer from efficient systematic uncertainty quantification. The Bayesian DNNs-based PFL is usually questioned of either over-simplified model structures or high computational and memory costs. In this paper, we introduce FedSI, a novel Bayesian DNNs-based subnetwork inference PFL framework. FedSI is simple and scalable by leveraging Bayesian methods to incorporate systematic uncertainties effectively. It implements a client-specific subnetwork inference mechanism, selects network parameters with large variance to be inferred through posterior distributions, and fixes the rest as deterministic ones. FedSI achieves fast and scalable inference while preserving the systematic uncertainties to the fullest extent. Extensive experiments on three different benchmark datasets demonstrate that FedSI outperforms existing Bayesian and non-Bayesian FL baselines in heterogeneous FL scenarios.

In this paper, we investigate the potential of reconfigurable intelligent surfaces (RISs) in facilitating passive/device-free three-dimensional (3D) drone localization within existing cellular infrastructure operating at millimeter-wave (mmWave) frequencies and employing multiple antennas at the transceivers. The developed localization system operates in the bi-static mode without requiring direct communication between the drone and the base station. We analyze the theoretical performance limits via Fisher information analysis and Cram\'er Rao lower bounds (CRLBs). Furthermore, we develop a low-complexity yet effective drone localization algorithm based on coordinate gradient descent and examine the impact of factors such as radar cross section (RCS) of the drone and training overhead on system performance. It is demonstrated that integrating RIS yields significant benefits over its RIS-free counterpart, as evidenced by both theoretical analyses and numerical simulations., Comment: 6 pages, 5 figures, submitted to IEEE GLOBECOM 2024

Future generations of mobile networks call for concurrent sensing and communication functionalities in the same hardware and/or spectrum. Compared to communication, sensing services often suffer from limited coverage, due to the high path loss of the reflected signal and the increased infrastructure requirements. To provide a more uniform quality of service, distributed multiple input multiple output (D-MIMO) systems deploy a large number of distributed nodes and efficiently control them, making distributed integrated sensing and communications (ISAC) possible. In this paper, we investigate ISAC in D-MIMO through the lens of different design architectures and deployments, revealing both conflicts and synergies. In addition, simulation and demonstration results reveal both opportunities and challenges towards the implementation of ISAC in D-MIMO.

The collective modes of the superconducting order parameter fluctuation can provide key insights into the nature of the superconductor. Recently, a family of superconductors has emerged in non-magnetic kagome material AV3Sb5 (A=K, Rb, Cs), exhibiting fertile emergent phenomenology. However, the collective behaviors of Cooper pairs have not been studied. Here, we report a distinct collective mode in CsV3-xTaxSb5 using scanning tunneling microscope/spectroscopy. The spectral line-shape is well-described by one isotropic and one anisotropic superconducting gap, and a bosonic mode due to electron-mode coupling. With increasing x, the two gaps move closer in energy, merge into two isotropic gaps of equal amplitude, and then increase synchronously. The mode energy decreases monotonically to well below 2{\Delta} and survives even after the charge density wave order is suppressed. We propose the interpretation of this collective mode as Leggett mode between different superconducting components or the Bardasis-Schrieffer mode due to a subleading superconducting component., Comment: 11 pages, 4 figures

Recently, the scale of transformers has grown rapidly, which introduces considerable challenges in terms of training overhead and inference efficiency in the scope of task adaptation. Existing works, namely Parameter-Efficient Fine-Tuning (PEFT) and model compression, have separately investigated the challenges. However, PEFT cannot guarantee the inference efficiency of the original backbone, especially for large-scale models. Model compression requires significant training costs for structure searching and re-training. Consequently, a simple combination of them cannot guarantee accomplishing both training efficiency and inference efficiency with minimal costs. In this paper, we propose a novel Parallel Yielding Re-Activation (PYRA) method for such a challenge of training-inference efficient task adaptation. PYRA first utilizes parallel yielding adaptive weights to comprehensively perceive the data distribution in downstream tasks. A re-activation strategy for token modulation is then applied for tokens to be merged, leading to calibrated token features. Extensive experiments demonstrate that PYRA outperforms all competing methods under both low compression rate and high compression rate, demonstrating its effectiveness and superiority in maintaining both training efficiency and inference efficiency for large-scale foundation models. Our code is available at https://github.com/THU-MIG/PYRA., Comment: 14 pages, 4 figures, Accepted by ECCV 2024

Personal digital data is a critical asset, and governments worldwide have enforced laws and regulations to protect data privacy. Data users have been endowed with the right to be forgotten of their data. In the course of machine learning (ML), the forgotten right requires a model provider to delete user data and its subsequent impact on ML models upon user requests. Machine unlearning emerges to address this, which has garnered ever-increasing attention from both industry and academia. While the area has developed rapidly, there is a lack of comprehensive surveys to capture the latest advancements. Recognizing this shortage, we conduct an extensive exploration to map the landscape of machine unlearning including the (fine-grained) taxonomy of unlearning algorithms under centralized and distributed settings, debate on approximate unlearning, verification and evaluation metrics, challenges and solutions for unlearning under different applications, as well as attacks targeting machine unlearning. The survey concludes by outlining potential directions for future research, hoping to serve as a guide for interested scholars.