Search

Foundation models (FMs) have shown remarkable advancements in enhancing the performance of intelligent applications. To address the need for data privacy in FM fine-tuning, federated learning has emerged as the de facto framework. Specifically, Federated FMs (FedFMs) fine-tuning using low-rank adaptation (LoRA) modules instead of the full model over multiple clients can achieve both parameter efficiency and data privacy. However, recent studies rarely address the challenges posed by clients with heterogeneous resources, particularly in GPU memory capacity. In this paper, we introduce Fed-piLot, an efficient FedFM fine-tuning framework with optimized local LoRA assignments for heterogeneous clients. By emphasizing the different memory consumption for training different LoRA layers, as well as the varying contributions of different layers to model performance, we formulate the LoRA assignment as a Knapsack Optimization Problem. We design a Local-Global Information Gain Score (IG-Score) based value function to optimize LoRA assignment under clients' memory constraints. To further mitigate the impact of heterogeneity in model updates, we propose a novel Spatial-Temporal model aggregation (STAgg) rule using the Dynamic Weight Adjustment (DWA) strategy. Experimental results on three datasets under both IID and non-IID conditions demonstrate the effectiveness and efficiency of Fed-piLot. The code will be publicly available.

Human Activity Recognition (HAR) has gained great attention from researchers due to the popularity of mobile devices and the need to observe users' daily activity data for better human-computer interaction. In this work, we collect a human activity recognition dataset called OPPOHAR consisting of phone IMU data. To facilitate the employment of HAR system in mobile phone and to achieve user-specific activity recognition, we propose a novel light-weight network called Non-stationary BERT with a two-stage training method. We also propose a simple yet effective data augmentation method to explore the deeper relationship between the accelerator and gyroscope data from the IMU. The network achieves the state-of-the-art performance testing on various activity recognition datasets and the data augmentation method demonstrates its wide applicability.

We survey the use of Chebyshev polynomials and Toeplitz theory for studying topological metamaterials. We consider both Hermitian and non-Hermitian systems of subwavelength resonators and provide a mathematical framework to explain some spectacular properties of metamaterials.

A fast multipole method (FMM)-accelerated surface integral equation (SIE) simulator, called XRL, is proposed for broadband resistance/inductance (RL) extraction under the magneto-quasi-static assumption. The proposed XRL has three key attributes that make it highly efficient and accurate for broadband RL extraction of complicated 3-D geometries: (i) The XRL leverages a novel centroid-midpoint basis transformation while discretizing surface currents, which allows converting edge-based vector potential computations to panel-based scalar potential computations. Such conversion makes the implementation of FMM straightforward and allows for drastically reducing the memory and computational time requirements of the simulator. (ii) The XRL employs a highly accurate equivalent surface impedance model that allows extracting RL parameters at low frequencies very accurately. (iii) The XRL makes use of a novel preconditioner, effectively including both diagonal entries and some near-field entries of the system matrix; such preconditioner significantly accelerates the iterative solution of SIE. The proposed XRL can accurately compute broadband RL parameters of arbitrarily shaped and large-scale structures on a desktop computer. It has been applied to RL parameter extraction of various practical structures, including two parallel square coils, a ball grid array (BGA) package and a high brand package on package. Its application to the parameter extraction of the BGA shows that the XRL requires 93.2x and 14.2x less computational time and memory resources compared to the commercial simulator Ansys Q3D for the same level of accuracy, respectively.

We revisit the calculation of the evolution equations for four unpolarized twist-3 transverse momentum dependent (TMD) parton distribution functions (PDFs) at $\mathcal {O}(\alpha_s)$. Unlike the existing calculations in the literature, which are based on the background field method, we derive the evolution equations directly through diagram expansion. Additionally, instead of using the $\delta$ regulator, we employ the exponential regulator to regularize the rapidity divergences. We find that the evolutions of the four twist-3 TMD PDFs are governed by eight homogeneous equations. Our evolution kernels agree with one set of results in the literature, but differ from another by a sign. Thus, our results help clarify the discrepancies between the evolution kernels reported in the literature. Moreover, considering the advantages of the exponential regulator in high-order perturbative calculations, our results are crucial for computing the twist-3 TMD PDFs at $\mathcal {O}(\alpha_s^2)$., Comment: 27 pages, 5 figures

Dataset distillation has emerged as a technique aiming to condense informative features from large, natural datasets into a compact and synthetic form. While recent advancements have refined this technique, its performance is bottlenecked by the prevailing class-specific synthesis paradigm. Under this paradigm, synthetic data is optimized exclusively for a pre-assigned one-hot label, creating an implicit class barrier in feature condensation. This leads to inefficient utilization of the distillation budget and oversight of inter-class feature distributions, which ultimately limits the effectiveness and efficiency, as demonstrated in our analysis. To overcome these constraints, this paper presents the Inter-class Feature Compensator (INFER), an innovative distillation approach that transcends the class-specific data-label framework widely utilized in current dataset distillation methods. Specifically, INFER leverages a Universal Feature Compensator (UFC) to enhance feature integration across classes, enabling the generation of multiple additional synthetic instances from a single UFC input. This significantly improves the efficiency of the distillation budget. Moreover, INFER enriches inter-class interactions during the distillation, thereby enhancing the effectiveness and generalizability of the distilled data. By allowing for the linear interpolation of labels similar to those in the original dataset, INFER meticulously optimizes the synthetic data and dramatically reduces the size of soft labels in the synthetic dataset to almost zero, establishing a new benchmark for efficiency and effectiveness in dataset distillation.

Recently, it has been observed that the Floquet-Bloch transform with real quasiperiodicities fails to capture the spectral properties of non-reciprocal systems. The aim of this paper is to introduce the notion of a generalised Brillouin zone by allowing the quasiperiodicities to be complex in order to rectify this. It is proved that this shift of the Brillouin zone into the complex plane accounts for the unidirectional spatial decay of the eigenmodes and leads to correct spectral convergence properties. The results in this paper clarify and prove rigorously how the spectral properties of a finite structure are associated with those of the corresponding semi-infinitely or infinitely periodic lattices and give explicit characterisations of how to extend the Hermitian theory to non-reciprocal settings. Based on our theory, we characterise the generalised Brillouin zone for both open boundary conditions and periodic boundary conditions. Our results are consistent with the physical literature and give explicit generalisations to the $k$-Toeplitz matrix cases.

This survey addresses the critical challenge of deepfake detection amidst the rapid advancements in artificial intelligence. As AI-generated media, including video, audio and text, become more realistic, the risk of misuse to spread misinformation and commit identity fraud increases. Focused on face-centric deepfakes, this work traces the evolution from traditional single-modality methods to sophisticated multi-modal approaches that handle audio-visual and text-visual scenarios. We provide comprehensive taxonomies of detection techniques, discuss the evolution of generative methods from auto-encoders and GANs to diffusion models, and categorize these technologies by their unique attributes. To our knowledge, this is the first survey of its kind. We also explore the challenges of adapting detection methods to new generative models and enhancing the reliability and robustness of deepfake detectors, proposing directions for future research. This survey offers a detailed roadmap for researchers, supporting the development of technologies to counter the deceptive use of AI in media creation, particularly facial forgery. A curated list of all related papers can be found at \href{https://github.com/qiqitao77/Comprehensive-Advances-in-Deepfake-Detection-Spanning-Diverse-Modalities}{https://github.com/qiqitao77/Awesome-Comprehensive-Deepfake-Detection}., Comment: P. Liu is with the Department of Computer Science and Engineering, University of Nevada, Reno, NV, 89512. Q. Tao and J. Zhou are with Centre for Frontier AI Research (CFAR), and Institute of High Performance Computing (IHPC), A*STAR, Singapore. J. Zhou is also with Centre for Advanced Technologies in Online Safety (CATOS), A*STAR, Singapore. J. Zhou is the corresponding author

Unlike Li-ion transport in the bulk of carbonaceous materials, little is known about Li-ion diffusion on their surface. In this study, we have discovered an ultra-fast Li-ion transport phenomenon on the surface of carbonaceous materials, particularly when they have limited Li insertion capacity along with a high surface area. This is exemplified by a carbon black, Ketjen Black (KB). An ionic conductivity of 18.1 mS cm-1 at room temperature is observed, far exceeding most solid-state ion conductors. Theoretical calculations reveal a low diffusion barrier for the surface Li species. The species is also identified as Li*, which features a partial positive charge. As a result, lithiated KB functions effectively as an interlayer between Li and solid-state electrolytes (SSE) to mitigate dendrite growth and cell shorting. This function is found to be electrolyte agnostic, effective for both sulfide and halide SSEs. Further, lithiated KB can act as a high-performance mixed ion/electron conductor that is thermodynamically stable at potentials near Li metal. A graphite anode mixed with KB instead of a solid electrolyte demonstrates full utilization with a capacity retention of ~85% over 300 cycles. The discovery of this surface-mediated ultra-fast Li-ion transport mechanism provides new directions for the design of solid-state ion conductors and solid-state batteries., Comment: 21 pages, 6 figures

The aim of this paper is to illustrate both analytically and numerically the interplay of two fundamentally distinct non-Hermitian mechanisms in a deep subwavelength regime. Considering a parity-time symmetric system of one-dimensional subwavelength resonators equipped with two kinds of non-Hermiticity - an imaginary gauge potential and on-site gain and loss - we prove that all but two eigenmodes of the system decouple when going through an exceptional point. By tuning the gain-to-loss ratio, the system changes from a phase with unbroken parity-time symmetry to a phase with broken parity-time symmetry. At the macroscopic level, this is observed as a transition from symmetrical eigenmodes to condensated eigenmodes at one edge of the structure. Mathematically, it arises from a topological state change. The results of this paper open the door to the justification of a variety of phenomena arising from the interplay between non-Hermitian reciprocal and non-reciprocal mechanisms not only in subwavelength wave physics but also in quantum mechanics where the tight binding model coupled with the nearest neighbour approximation can be analysed with the same tools as those developed here.

Dataset distillation is an advanced technique aimed at compressing datasets into significantly smaller counterparts, while preserving formidable training performance. Significant efforts have been devoted to promote evaluation accuracy under limited compression ratio while overlooked the robustness of distilled dataset. In this work, we introduce a comprehensive benchmark that, to the best of our knowledge, is the most extensive to date for evaluating the adversarial robustness of distilled datasets in a unified way. Our benchmark significantly expands upon prior efforts by incorporating a wider range of dataset distillation methods, including the latest advancements such as TESLA and SRe2L, a diverse array of adversarial attack methods, and evaluations across a broader and more extensive collection of datasets such as ImageNet-1K. Moreover, we assessed the robustness of these distilled datasets against representative adversarial attack algorithms like PGD and AutoAttack, while exploring their resilience from a frequency perspective. We also discovered that incorporating distilled data into the training batches of the original dataset can yield to improvement of robustness., Comment: * denotes equal contributions; ^ denotes corresponding author. In this updated version, we have expanded our research to include more experiments on various adversarial attack methods and latest dataset distillation studies. All new results have been incorporated into the document

This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images despite limited training data. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. However, the rapid advancements in synthesis technology have led to specific artifacts for each generation model. Consequently, these detectors have exhibited a lack of proficiency in learning the frequency domain and tend to overfit to the artifacts present in the training data, leading to suboptimal performance on unseen sources. To address this issue, we introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors. Our method forces the detector to continuously focus on high-frequency information, exploiting high-frequency representation of features across spatial and channel dimensions. Additionally, we incorporate a straightforward frequency domain learning module to learn source-agnostic features. It involves convolutional layers applied to both the phase spectrum and amplitude spectrum between the Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (iFFT). Extensive experimentation involving 17 GANs demonstrates the effectiveness of our proposed method, showcasing state-of-the-art performance (+9.8\%) while requiring fewer parameters. The code is available at {\cred \url{https://github.com/chuangchuangtan/FreqNet-DeepfakeDetection}}., Comment: 9 pages, 4 figures, AAAI24

Recently, the proliferation of increasingly realistic synthetic images generated by various generative adversarial networks has increased the risk of misuse. Consequently, there is a pressing need to develop a generalizable detector for accurately recognizing fake images. The conventional methods rely on generating diverse training sources or large pretrained models. In this work, we show that, on the contrary, the small and training-free filter is sufficient to capture more general artifact representations. Due to its unbias towards both the training and test sources, we define it as Data-Independent Operator (DIO) to achieve appealing improvements on unseen sources. In our framework, handcrafted filters and the randomly-initialized convolutional layer can be used as the training-free artifact representations extractor with excellent results. With the data-independent operator of a popular classifier, such as Resnet50, one could already reach a new state-of-the-art without bells and whistles. We evaluate the effectiveness of the DIO on 33 generation models, even DALLE and Midjourney. Our detector achieves a remarkable improvement of $13.3\%$, establishing a new state-of-the-art performance. The DIO and its extension can serve as strong baselines for future methods. The code is available at \url{https://github.com/chuangchuangtan/Data-Independent-Operator}., Comment: 12 pages, 3 figures

Web-scale search systems typically tackle the scalability challenge with a two-step paradigm: retrieval and ranking. The retrieval step, also known as candidate selection, often involves extracting standardized entities, creating an inverted index, and performing term matching for retrieval. Such traditional methods require manual and time-consuming development of query models. In this paper, we discuss applying learning-to-retrieve technology to enhance LinkedIns job search and recommendation systems. In the realm of promoted jobs, the key objective is to improve the quality of applicants, thereby delivering value to recruiter customers. To achieve this, we leverage confirmed hire data to construct a graph that evaluates a seeker's qualification for a job, and utilize learned links for retrieval. Our learned model is easy to explain, debug, and adjust. On the other hand, the focus for organic jobs is to optimize seeker engagement. We accomplished this by training embeddings for personalized retrieval, fortified by a set of rules derived from the categorization of member feedback. In addition to a solution based on a conventional inverted index, we developed an on-GPU solution capable of supporting both KNN and term matching efficiently.

We present LinkSAGE, an innovative framework that integrates Graph Neural Networks (GNNs) into large-scale personalized job matching systems, designed to address the complex dynamics of LinkedIns extensive professional network. Our approach capitalizes on a novel job marketplace graph, the largest and most intricate of its kind in industry, with billions of nodes and edges. This graph is not merely extensive but also richly detailed, encompassing member and job nodes along with key attributes, thus creating an expansive and interwoven network. A key innovation in LinkSAGE is its training and serving methodology, which effectively combines inductive graph learning on a heterogeneous, evolving graph with an encoder-decoder GNN model. This methodology decouples the training of the GNN model from that of existing Deep Neural Nets (DNN) models, eliminating the need for frequent GNN retraining while maintaining up-to-date graph signals in near realtime, allowing for the effective integration of GNN insights through transfer learning. The subsequent nearline inference system serves the GNN encoder within a real-world setting, significantly reducing online latency and obviating the need for costly real-time GNN infrastructure. Validated across multiple online A/B tests in diverse product scenarios, LinkSAGE demonstrates marked improvements in member engagement, relevance matching, and member retention, confirming its generalizability and practical impact.

In this paper, we present LiGNN, a deployed large-scale Graph Neural Networks (GNNs) Framework. We share our insight on developing and deployment of GNNs at large scale at LinkedIn. We present a set of algorithmic improvements to the quality of GNN representation learning including temporal graph architectures with long term losses, effective cold start solutions via graph densification, ID embeddings and multi-hop neighbor sampling. We explain how we built and sped up by 7x our large-scale training on LinkedIn graphs with adaptive sampling of neighbors, grouping and slicing of training data batches, specialized shared-memory queue and local gradient optimization. We summarize our deployment lessons and learnings gathered from A/B test experiments. The techniques presented in this work have contributed to an approximate relative improvements of 1% of Job application hearing back rate, 2% Ads CTR lift, 0.5% of Feed engaged daily active users, 0.2% session lift and 0.1% weekly active user lift from people recommendation. We believe that this work can provide practical solutions and insights for engineers who are interested in applying Graph neural networks at large scale.

Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have raised growing concerns. As one of the most promising directions, graph condensation methods address these issues by employing gradient matching, aiming to condense the full graph into a more concise yet information-rich synthetic set. Though encouraging, these strategies primarily emphasize matching directions of the gradients, which leads to deviations in the training trajectories. Such deviations are further magnified by the differences between the condensation and evaluation phases, culminating in accumulated errors, which detrimentally affect the performance of the condensed graphs. In light of this, we propose a novel graph condensation method named \textbf{C}raf\textbf{T}ing \textbf{R}ationa\textbf{L} trajectory (\textbf{CTRL}), which offers an optimized starting point closer to the original dataset's feature distribution and a more refined strategy for gradient matching. Theoretically, CTRL can effectively neutralize the impact of accumulated errors on the performance of condensed graphs. We provide extensive experiments on various graph datasets and downstream tasks to support the effectiveness of CTRL. Code is released at https://github.com/NUS-HPC-AI-Lab/CTRL., Comment: An effective method for graph condensation

We establish new results on the spectra and pseudo-spectra of tridiagonal $k$-Toeplitz operators and matrices. In particular, we prove the connection between the winding number of the eigenvalues of the symbol function and the exponential decay of the associated eigenvectors (or pseudo-eigenvectors). Our results elucidate the topological origin of the non-Hermitian skin effect in general one-dimensional polymer systems of subwavelength resonators with imaginary gauge potentials, proving the observation and conjecture in arXiv:2307.13551. We also numerically verify our theory for these systems., Comment: 18 pages, 6 figures

Existing text-to-image editing methods tend to excel either in rigid or non-rigid editing but encounter challenges when combining both, resulting in misaligned outputs with the provided text prompts. In addition, integrating reference images for control remains challenging. To address these issues, we present a versatile image editing framework capable of executing both rigid and non-rigid edits, guided by either textual prompts or reference images. We leverage a dual-path injection scheme to handle diverse editing scenarios and introduce an integrated self-attention mechanism for fusion of appearance and structural information. To mitigate potential visual artifacts, we further employ latent fusion techniques to adjust intermediate latents. Compared to previous work, our approach represents a significant advance in achieving precise and versatile image editing. Comprehensive experiments validate the efficacy of our method, showcasing competitive or superior results in text-based editing and appearance transfer tasks, encompassing both rigid and non-rigid settings., Comment: 15 pages, 13 figures

Recently, the proliferation of highly realistic synthetic images, facilitated through a variety of GANs and Diffusions, has significantly heightened the susceptibility to misuse. While the primary focus of deepfake detection has traditionally centered on the design of detection algorithms, an investigative inquiry into the generator architectures has remained conspicuously absent in recent years. This paper contributes to this lacuna by rethinking the architectures of CNN-based generators, thereby establishing a generalized representation of synthetic artifacts. Our findings illuminate that the up-sampling operator can, beyond frequency-based artifacts, produce generalized forgery artifacts. In particular, the local interdependence among image pixels caused by upsampling operators is significantly demonstrated in synthetic images generated by GAN or diffusion. Building upon this observation, we introduce the concept of Neighboring Pixel Relationships(NPR) as a means to capture and characterize the generalized structural artifacts stemming from up-sampling operations. A comprehensive analysis is conducted on an open-world dataset, comprising samples generated by \tft{28 distinct generative models}. This analysis culminates in the establishment of a novel state-of-the-art performance, showcasing a remarkable \tft{11.6\%} improvement over existing methods. The code is available at https://github.com/chuangchuangtan/NPR-DeepfakeDetection., Comment: 10 pages, 4 figures

This paper studies wave localisation in chains of finitely many resonators. There is an extensive theory predicting the existence of localised modes induced by defects in infinitely periodic systems. This work extends these principles to finite-sized systems. We consider finite systems of subwavelength resonators arranged in dimers that have a geometric defect in the structure. This is a classical wave analogue of the Su-Schrieffer-Heeger model. We prove the existence of a spectral gap for defectless finite dimer structures and find a direct relationship between eigenvalues being within the spectral gap and the localisation of their associated eigenmode. Then we show the existence and uniqueness of an eigenvalue in the gap in the defect structure, proving the existence of a unique localised interface mode. To the best of our knowledge, our method, based on Chebyshev polynomials, is the first to characterise quantitatively the localised interface modes in systems of finitely many resonators., Comment: 21 pages

Vehicle Re-identification (Re-ID) has been broadly studied in the last decade; however, the different camera view angle leading to confused discrimination in the feature subspace for the vehicles of various poses, is still challenging for the Vehicle Re-ID models in the real world. To promote the Vehicle Re-ID models, this paper proposes to synthesize a large number of vehicle images in the target pose, whose idea is to project the vehicles of diverse poses into the unified target pose so as to enhance feature discrimination. Considering that the paired data of the same vehicles in different traffic surveillance cameras might be not available in the real world, we propose the first Pair-flexible Pose Guided Image Synthesis method for Vehicle Re-ID, named as VehicleGAN in this paper, which works for both supervised and unsupervised settings without the knowledge of geometric 3D models. Because of the feature distribution difference between real and synthetic data, simply training a traditional metric learning based Re-ID model with data-level fusion (i.e., data augmentation) is not satisfactory, therefore we propose a new Joint Metric Learning (JML) via effective feature-level fusion from both real and synthetic data. Intensive experimental results on the public VeRi-776 and VehicleID datasets prove the accuracy and effectiveness of our proposed VehicleGAN and JML.

We study the non-Hermitian skin effect in a three-dimensional system of finitely many subwavelength resonators with an imaginary gauge potential. We introduce a discrete approximation of the eigenmodes and eigenfrequencies of the system in terms of the eigenvectors and eigenvalues of the so-called gauge capacitance matrix $\mathcal{C}_N^\gamma$, which is a dense matrix due to long-range interactions in the system. Based on translational invariance of this matrix and the decay of its off-diagonal entries, we prove the condensation of the eigenmodes at one edge of the structure by showing the exponential decay of its pseudo-eigenvectors. In particular, we consider a range-k approximation to keep the long-range interaction to a certain extent, thus obtaining a k-banded gauge capacitance matrix $\mathcal{C}_{N,k}^\gamma$ . Using techniques for Toeplitz matrices and operators, we establish the exponential decay of the pseudo-eigenvectors of $\mathcal{C}_{N,k}^\gamma$ and demonstrate that they approximate those of the gauge capacitance matrix $\mathcal{C}_N^\gamma$ well. Our results are numerically verified. In particular, we show that long-range interactions affect only the first eigenmodes in the system. As a result, a tridiagonal approximation of the gauge capacitance matrix, similar to the nearest-neighbour approximation in quantum mechanics, provides a good approximation for the higher modes. Moreover, we also illustrate numerically the behaviour of the eigenmodes and the stability of the non-Hermitian skin effect with respect to disorder in a variety of three-dimensional structures., Comment: 32 pages, 11 figures