39 results on '"Avestimehr, Salman"'
Search Results
2. Achieving small-batch accuracy with large-batch scalability via Hessian-aware learning rate adjustment
- Author
-
Lee, Sunwoo, He, Chaoyang, and Avestimehr, Salman
- Published
- 2023
- Full Text
- View/download PDF
3. Partial model averaging in Federated Learning: Performance guarantees and benefits
- Author
-
Lee, Sunwoo, Sahu, Anit Kumar, He, Chaoyang, and Avestimehr, Salman
- Published
- 2023
- Full Text
- View/download PDF
4. FedMultimodal: A Benchmark For Multimodal Federated Learning
- Author
-
Feng, Tiantian, Bose, Digbalay, Zhang, Tuo, Hebbar, Rajat, Ramakrishna, Anil, Gupta, Rahul, Zhang, Mi, Avestimehr, Salman, and Narayanan, Shrikanth
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Distributed, Parallel, and Cluster Computing ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal., This paper was accepted to KDD 2023 Applied Data Science (ADS) track
- Published
- 2023
5. FedMLSecurity: A Benchmark for Attacks and Defenses in Federated Learning and LLMs
- Author
-
Han, Shanshan, Buyukates, Baturalp, Hu, Zijian, Jin, Han, Jin, Weizhao, Sun, Lichao, Wang, Xiaoyang, Xie, Chulin, Zhang, Kai, Zhang, Qifan, Zhang, Yuhui, He, Chaoyang, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Artificial Intelligence (cs.AI) ,Computer Science - Cryptography and Security ,Computer Science - Artificial Intelligence ,Cryptography and Security (cs.CR) - Abstract
This paper introduces FedMLSecurity, a benchmark that simulates adversarial attacks and corresponding defense mechanisms in Federated Learning (FL). As an integral module of the open-sourced library FedML that facilitates FL algorithm development and performance comparison, FedMLSecurity enhances the security assessment capacity of FedML. FedMLSecurity comprises two principal components: FedMLAttacker, which simulates attacks injected into FL training, and FedMLDefender, which emulates defensive strategies designed to mitigate the impacts of the attacks. FedMLSecurity is open-sourced 1 and is customizable to a wide range of machine learning models (e.g., Logistic Regression, ResNet, GAN, etc.) and federated optimizers (e.g., FedAVG, FedOPT, FedNOVA, etc.). Experimental evaluations in this paper also demonstrate the ease of application of FedMLSecurity to Large Language Models (LLMs), further reinforcing its versatility and practical utility in various scenarios.
- Published
- 2023
6. GPT-FL: Generative Pre-trained Model-Assisted Federated Learning
- Author
-
Zhang, Tuo, Feng, Tiantian, Alam, Samiul, Dimitriadis, Dimitrios, Zhang, Mi, Narayanan, Shrikanth S., and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Distributed, Parallel, and Cluster Computing ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
In this work, we propose GPT-FL, a generative pre-trained model-assisted federated learning (FL) framework. At its core, GPT-FL leverages generative pre-trained models to generate diversified synthetic data. These generated data are used to train a downstream model on the server, which is then fine-tuned with private client data under the standard FL framework. We show that GPT-FL consistently outperforms state-of-the-art FL methods in terms of model test accuracy, communication efficiency, and client sampling efficiency. Through comprehensive ablation analysis, we discover that the downstream model generated by synthetic data plays a crucial role in controlling the direction of gradient diversity during FL training, which enhances convergence speed and contributes to the notable accuracy boost observed with GPT-FL. Also, regardless of whether the target data falls within or outside the domain of the pre-trained generative model, GPT-FL consistently achieves significant performance gains, surpassing the results obtained by models trained solely with FL or synthetic data.
- Published
- 2023
7. Federated Learning for Clients' Data Privacy Assurance in Food Service Industry.
- Author
-
Taheri Gorji, Hamed, Saeedi, Mahdi, Mushtaq, Erum, Kashani Zadeh, Hossein, Husarik, Kaylee, Shahabi, Seyed Mojtaba, Qin, Jianwei, Chan, Diane E., Baek, Insuck, Kim, Moon S., Akhbardeh, Alireza, Sokolov, Stanislav, Avestimehr, Salman, MacKinnon, Nicholas, Vasefi, Fartash, and Tavakolian, Kouhyar
- Subjects
DATA privacy ,DEEP learning ,MACHINE learning ,FOOD service ,ASSURANCE services ,FOOD industry - Abstract
The food service industry must ensure that service facilities are free of foodborne pathogens hosted by organic residues and biofilms. Foodborne diseases put customers at risk and compromise the reputations of service providers. Fluorescence imaging, empowered by state-of-the-art artificial intelligence (AI) algorithms, can detect invisible residues. However, using AI requires large datasets that are most effective when collected from actual users, raising concerns about data privacy and possible leakage of sensitive information. In this study, we employed a decentralized privacy-preserving technology to address client data privacy issues. When federated learning (FL) is used, there is no need for data sharing across clients or data centralization on a server. We used FL and a new fluorescence imaging technology and applied two deep learning models, MobileNetv3 and DeepLabv3+, to identify and segment invisible residues on food preparation equipment and surfaces. We used FedML as our FL framework and Fedavg as the aggregation algorithm. The model achieved training and testing accuracies of 95.83% and 94.94% for classification between clean and contamination frames, respectively, and resulted in intersection over union (IoU) scores of 91.23% and 89.45% for training and testing, respectively, of segmentation of the contaminated areas. The results demonstrated that using federated learning combined with fluorescence imaging and deep learning algorithms can improve the performance of cleanliness auditing systems while assuring client data privacy. [ABSTRACT FROM AUTHOR]
- Published
- 2023
- Full Text
- View/download PDF
8. Federated Alternate Training (FAT): Leveraging Unannotated Data Silos in Federated Segmentation for Medical Imaging
- Author
-
Mushtaq, Erum, Bakman, Yavuz Faruk, Ding, Jie, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Vision and Pattern Recognition (cs.CV) ,FOS: Biological sciences ,Computer Science - Computer Vision and Pattern Recognition ,Quantitative Biology - Quantitative Methods ,Quantitative Methods (q-bio.QM) ,Machine Learning (cs.LG) - Abstract
Federated Learning (FL) aims to train a machine learning (ML) model in a distributed fashion to strengthen data privacy with limited data migration costs. It is a distributed learning framework naturally suitable for privacy-sensitive medical imaging datasets. However, most current FL-based medical imaging works assume silos have ground truth labels for training. In practice, label acquisition in the medical field is challenging as it often requires extensive labor and time costs. To address this challenge and leverage the unannotated data silos to improve modeling, we propose an alternate training-based framework, Federated Alternate Training (FAT), that alters training between annotated data silos and unannotated data silos. Annotated data silos exploit annotations to learn a reasonable global segmentation model. Meanwhile, unannotated data silos use the global segmentation model as a target model to generate pseudo labels for self-supervised learning. We evaluate the performance of the proposed framework on two naturally partitioned Federated datasets, KiTS19 and FeTS2021, and show its promising performance., Camera Ready Version of ISBI2023 Accepted work
- Published
- 2023
9. TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training
- Author
-
Zhang, Tuo, Gao, Lei, Lee, Sunwoo, Zhang, Mi, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Distributed, Parallel, and Cluster Computing ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
In cross-device Federated Learning (FL) environments, scaling synchronous FL methods is challenging as stragglers hinder the training process. Moreover, the availability of each client to join the training is highly variable over time due to system heterogeneities and intermittent connectivity. Recent asynchronous FL methods (e.g., FedBuff) have been proposed to overcome these issues by allowing slower users to continue their work on local training based on stale models and to contribute to aggregation when ready. However, we show empirically that this method can lead to a substantial drop in training accuracy as well as a slower convergence rate. The primary reason is that fast-speed devices contribute to many more rounds of aggregation while others join more intermittently or not at all, and with stale model updates. To overcome this barrier, we propose TimelyFL, a heterogeneity-aware asynchronous FL framework with adaptive partial training. During the training, TimelyFL adjusts the local training workload based on the real-time resource capabilities of each client, aiming to allow more available clients to join in the global update without staleness. We demonstrate the performance benefits of TimelyFL by conducting extensive experiments on various datasets (e.g., CIFAR-10, Google Speech, and Reddit) and models (e.g., ResNet20, VGG11, and ALBERT). In comparison with the state-of-the-art (i.e., FedBuff), our evaluations reveal that TimelyFL improves participation rate by 21.13%, harvests 1.28x - 2.89x more efficiency on convergence rate, and provides a 6.25% increment on test accuracy.
- Published
- 2023
10. Secure Federated Learning against Model Poisoning Attacks via Client Filtering
- Author
-
Yaldiz, Duygu Nur, Zhang, Tuo, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Cryptography and Security ,Computer Science - Distributed, Parallel, and Cluster Computing ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Cryptography and Security (cs.CR) - Abstract
Given the distributed nature, detecting and defending against the backdoor attack under federated learning (FL) systems is challenging. In this paper, we observe that the cosine similarity of the last layer's weight between the global model and each local update could be used effectively as an indicator of malicious model updates. Therefore, we propose CosDefense, a cosine-similarity-based attacker detection algorithm. Specifically, under CosDefense, the server calculates the cosine similarity score of the last layer's weight between the global model and each client update, labels malicious clients whose score is much higher than the average, and filters them out of the model aggregation in each round. Compared to existing defense schemes, CosDefense does not require any extra information besides the received model updates to operate and is compatible with client sampling. Experiment results on three real-world datasets demonstrate that CosDefense could provide robust performance under the state-of-the-art FL poisoning attack.
- Published
- 2023
11. FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning System
- Author
-
Jin, Weizhao, Yao, Yuhang, Han, Shanshan, Joe-Wong, Carlee, Ravi, Srivatsan, Avestimehr, Salman, and He, Chaoyang
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Cryptography and Security ,Cryptography and Security (cs.CR) ,Machine Learning (cs.LG) - Abstract
Federated Learning (FL) enables machine learning model training on distributed edge devices by aggregating local model updates rather than local data. However, privacy concerns arise as the FL server's access to local model updates can potentially reveal sensitive personal information by performing attacks like gradient inversion recovery. To address these concerns, privacy-preserving methods, such as Homomorphic Encryption (HE)-based approaches, have been proposed. Despite HE's post-quantum security advantages, its applications suffer from impractical overheads. In this paper, we present FedML-HE, the first practical system for efficient HE-based secure federated aggregation that provides a user/device-friendly deployment platform. FL-HE utilizes a novel universal overhead optimization scheme, significantly reducing both computation and communication overheads during deployment while providing customizable privacy guarantees. Our optimized system demonstrates considerable overhead reduction, particularly for large models (e.g., ~10x reduction for HE-federated training of ResNet-50 and ~40x reduction for BERT), demonstrating the potential for scalable HE-based FL deployment.
- Published
- 2023
12. FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training
- Author
-
Tang, Zhenheng, Chu, Xiaowen, Ran, Ryan Yide, Lee, Sunwoo, Shi, Shaohuai, Zhang, Yonggang, Wang, Yuxin, Liang, Alex Qiaozhong, Avestimehr, Salman, and He, Chaoyang
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Distributed, Parallel, and Cluster Computing ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
Federated Learning (FL) enables collaborations among clients for train machine learning models while protecting their data privacy. Existing FL simulation platforms that are designed from the perspectives of traditional distributed training, suffer from laborious code migration between simulation and production, low efficiency, low GPU utility, low scalability with high hardware requirements and difficulty of simulating stateful clients. In this work, we firstly demystify the challenges and bottlenecks of simulating FL, and design a new FL system named as FedML \texttt{Parrot}. It improves the training efficiency, remarkably relaxes the requirements on the hardware, and supports efficient large-scale FL experiments with stateful clients by: (1) sequential training clients on devices; (2) decomposing original aggregation into local and global aggregation on devices and server respectively; (3) scheduling tasks to mitigate straggler problems and enhance computing utility; (4) distributed client state manager to support various FL algorithms. Besides, built upon our generic APIs and communication interfaces, users can seamlessly transform the simulation into the real-world deployment without modifying codes. We evaluate \texttt{Parrot} through extensive experiments for training diverse models on various FL datasets to demonstrate that \texttt{Parrot} can achieve simulating over 1000 clients (stateful or stateless) with flexible GPU devices setting ($4 \sim 32$) and high GPU utility, 1.2 $\sim$ 4 times faster than FedScale, and 10 $\sim$ 100 times memory saving than FedML. And we verify that \texttt{Parrot} works well with homogeneous and heterogeneous devices in three different clusters. Two FL algorithms with stateful clients and four algorithms with stateless clients are simulated to verify the wide adaptability of \texttt{Parrot} to different algorithms.
- Published
- 2023
13. FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
- Author
-
Terrail, Jean Ogier du, Ayed, Samy-Safwan, Cyffers, Edwige, Grimberg, Felix, He, Chaoyang, Loeb, Regis, Mangold, Paul, Marchand, Tanguy, Marfoq, Othmane, Mushtaq, Erum, Muzellec, Boris, Philippenko, Constantin, Silva, Santiago, Teleńczuk, Maria, Albarqouni, Shadi, Avestimehr, Salman, Bellet, Aurélien, Dieuleveut, Aymeric, Jaggi, Martin, Karimireddy, Sai Praneeth, Lorenzi, Marco, Neglia, Giovanni, Tommasi, Marc, Andreux, Mathieu, Owkin France, Université Côte d'Azur (UCA), Machine Learning in Information Networks (MAGNET), Inria Lille - Nord Europe, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 (CRIStAL), Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS), Ecole Polytechnique Fédérale de Lausanne (EPFL), FedML, Inc (FedML), Network Engineering and Operations (NEO ), Inria Sophia Antipolis - Méditerranée (CRISAM), Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), University of Southern California (USC), Centre de Mathématiques Appliquées - Ecole Polytechnique (CMAP), École polytechnique (X)-Centre National de la Recherche Scientifique (CNRS), University Hospital Bonn, Helmholtz Munich, Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), University of California [Berkeley] (UC Berkeley), University of California (UC), E-Patient : Images, données & mOdèles pour la médeciNe numériquE (EPIONE), ANR-20-CE23-0015,PRIDE,Apprentissage automatique décentralisé et préservant la vie privée(2020), Tommasi, Marc, and Apprentissage automatique décentralisé et préservant la vie privée - - PRIDE2020 - ANR-20-CE23-0015 - AAPG2020 - VALID
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,[INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG] ,Computer Vision and Pattern Recognition (cs.CV) ,Computer Science - Computer Vision and Pattern Recognition ,[INFO.INFO-LG] Computer Science [cs]/Machine Learning [cs.LG] ,Machine Learning (cs.LG) - Abstract
Federated Learning (FL) is a novel approach enabling several clients holding sensitive data to collaboratively train machine learning models, without centralizing data. The cross-silo FL setting corresponds to the case of few ($2$--$50$) reliable clients, each holding medium to large datasets, and is typically found in applications such as healthcare, finance, or industry. While previous works have proposed representative datasets for cross-device FL, few realistic healthcare cross-silo FL datasets exist, thereby slowing algorithmic research in this critical application. In this work, we propose a novel cross-silo dataset suite focused on healthcare, FLamby (Federated Learning AMple Benchmark of Your cross-silo strategies), to bridge the gap between theory and practice of cross-silo FL. FLamby encompasses 7 healthcare datasets with natural splits, covering multiple tasks, modalities, and data volumes, each accompanied with baseline training code. As an illustration, we additionally benchmark standard FL algorithms on all datasets. Our flexible and modular suite allows researchers to easily download datasets, reproduce results and re-use the different components for their research. FLamby is available at~\url{www.github.com/owkin/flamby}., Accepted to NeurIPS, Datasets and Benchmarks Track, this version fixes typos in the datasets' table and the appendix
- Published
- 2022
14. Federated Learning of Large Models at the Edge via Principal Sub-Model Training
- Author
-
Niu, Yue, Prakash, Saurav, Kundu, Souvik, Lee, Sunwoo, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Artificial Intelligence (cs.AI) ,Computer Science - Artificial Intelligence ,Machine Learning (cs.LG) - Abstract
Limited compute, memory, and communication capabilities of edge users create a significant bottleneck for federated learning (FL) of large models. Current literature typically tackles the challenge with a heterogeneous client setting or allows training to be offloaded to the server. However, the former requires a fraction of clients to train near-full models, which may not be achievable at the edge; while the latter can compromise privacy with sharing of intermediate representations or labels. In this work, we consider a realistic, but much less explored, cross-device FL setting in which no client has the capacity to train a full large model nor is willing to share any intermediate representations with the server. To this end, we present Principal Sub-Model (PriSM) training methodology, which leverages models low-rank structure and kernel orthogonality to train sub-models in the orthogonal kernel space. More specifically, by applying singular value decomposition to original kernels in the server model, PriSM first obtains a set of principal orthogonal kernels with importance weighed by their singular values. Thereafter, PriSM utilizes a novel sampling strategy that selects different subsets of the principal kernels independently to create sub-models for clients with reduced computation and communication requirements. Importantly, a kernel with a large singular value is assigned with a high sampling probability. Thus, each sub-model is a low-rank approximation of the full large model, and all clients together achieve nearly full coverage of the principal kernels. To further improve memory efficiency, PriSM exploits low-rank structure in intermediate representations and allows each sub-model to learn only a subset of them while still preserving training performance.
- Published
- 2022
15. How Much Privacy Does Federated Learning with Secure Aggregation Guarantee?
- Author
-
Elkordy, Ahmed Roushdy, Zhang, Jiang, Ezzeldin, Yahya H., Psounis, Konstantinos, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Cryptography and Security ,Information Theory (cs.IT) ,Computer Science - Information Theory ,General Earth and Planetary Sciences ,Cryptography and Security (cs.CR) ,Machine Learning (cs.LG) ,General Environmental Science - Abstract
Federated learning (FL) has attracted growing interest for enabling privacy-preserving machine learning on data stored at multiple users while avoiding moving the data off-device. However, while data never leaves users' devices, privacy still cannot be guaranteed since significant computations on users' training data are shared in the form of trained local models. These local models have recently been shown to pose a substantial privacy threat through different privacy attacks such as model inversion attacks. As a remedy, Secure Aggregation (SA) has been developed as a framework to preserve privacy in FL, by guaranteeing the server can only learn the global aggregated model update but not the individual model updates. While SA ensures no additional information is leaked about the individual model update beyond the aggregated model update, there are no formal guarantees on how much privacy FL with SA can actually offer; as information about the individual dataset can still potentially leak through the aggregated model computed at the server. In this work, we perform a first analysis of the formal privacy guarantees for FL with SA. Specifically, we use Mutual Information (MI) as a quantification metric and derive upper bounds on how much information about each user's dataset can leak through the aggregated model update. When using the FedSGD aggregation algorithm, our theoretical bounds show that the amount of privacy leakage reduces linearly with the number of users participating in FL with SA. To validate our theoretical bounds, we use an MI Neural Estimator to empirically evaluate the privacy leakage under different FL setups on both the MNIST and CIFAR10 datasets. Our experiments verify our theoretical bounds for FedSGD, which show a reduction in privacy leakage as the number of users and local batch size grow, and an increase in privacy leakage with the number of training rounds., Accepted to appear in Proceedings on Privacy Enhancing Technologies (PoPETs) 2023
- Published
- 2022
16. Toward a Geometrical Understanding of Self-supervised Contrastive Learning
- Author
-
Cosentino, Romain, Sengupta, Anirvan, Avestimehr, Salman, Soltanolkotabi, Mahdi, Ortega, Antonio, Willke, Ted, and Tepper, Mariano
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Artificial Intelligence (cs.AI) ,Computer Science - Artificial Intelligence ,Machine Learning (cs.LG) - Abstract
Self-supervised learning (SSL) is currently one of the premier techniques to create data representations that are actionable for transfer learning in the absence of human annotations. Despite their success, the underlying geometry of these representations remains elusive, which obfuscates the quest for more robust, trustworthy, and interpretable models. In particular, mainstream SSL techniques rely on a specific deep neural network architecture with two cascaded neural networks: the encoder and the projector. When used for transfer learning, the projector is discarded since empirical results show that its representation generalizes more poorly than the encoder's. In this paper, we investigate this curious phenomenon and analyze how the strength of the data augmentation policies affects the data embedding. We discover a non-trivial relation between the encoder, the projector, and the data augmentation strength: with increasingly larger augmentation policies, the projector, rather than the encoder, is more strongly driven to become invariant to the augmentations. It does so by eliminating crucial information about the data by learning to project it into a low-dimensional space, a noisy estimate of the data manifold tangent plane in the encoder representation. This analysis is substantiated through a geometrical perspective with theoretical and empirical results.
- Published
- 2022
17. Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits
- Author
-
Lee, Sunwoo, Sahu, Anit Kumar, He, Chaoyang, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Statistics - Machine Learning ,Machine Learning (stat.ML) ,Machine Learning (cs.LG) - Abstract
Local Stochastic Gradient Descent (SGD) with periodic model averaging (FedAvg) is a foundational algorithm in Federated Learning. The algorithm independently runs SGD on multiple workers and periodically averages the model across all the workers. When local SGD runs with many workers, however, the periodic averaging causes a significant model discrepancy across the workers making the global loss converge slowly. While recent advanced optimization methods tackle the issue focused on non-IID settings, there still exists the model discrepancy issue due to the underlying periodic model averaging. We propose a partial model averaging framework that mitigates the model discrepancy issue in Federated Learning. The partial averaging encourages the local models to stay close to each other on parameter space, and it enables to more effectively minimize the global loss. Given a fixed number of iterations and a large number of workers (128), the partial averaging achieves up to 2.2% higher validation accuracy than the periodic full averaging.
- Published
- 2022
18. SPIDER: Searching Personalized Neural Architecture for Federated Learning
- Author
-
Mushtaq, Erum, He, Chaoyang, Ding, Jie, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Vision and Pattern Recognition (cs.CV) ,Computer Science - Computer Vision and Pattern Recognition ,Machine Learning (cs.LG) - Abstract
Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, in this work, we introduce SPIDER, an algorithmic framework that aims to Search Personalized neural architecture for federated learning. SPIDER is designed based on two unique features: (1) alternately optimizing one architecture-homogeneous global model (Supernet) in a generic FL manner and one architecture-heterogeneous local model that is connected to the global model by weight sharing-based regularization (2) achieving architecture-heterogeneous local model by a novel neural architecture search (NAS) method that can select optimal subnet progressively using operation-level perturbation on the accuracy value as the criterion. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods, and the searched personalized architectures are more inference efficient.
- Published
- 2021
19. SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
- Author
-
He, Chaoyang, Yang, Zhengyu, Mushtaq, Erum, Lee, Sunwoo, Soltanolkotabi, Mahdi, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Artificial Intelligence (cs.AI) ,Computer Science - Artificial Intelligence ,Machine Learning (cs.LG) - Abstract
Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy. An essential but rarely studied challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL compared to centralized training due to the fact that FL users are often reluctant to label their private data. Furthermore, due to the heterogeneous nature of the data at edge devices, it is crucial to develop personalized models. In this paper we propose self-supervised federated learning (SSFL), a unified self-supervised and personalized federated learning framework, and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we demonstrate that the standard FedAvg algorithm is compatible with recent breakthroughs in centralized self-supervised learning such as SimSiam networks. Moreover, to deal with data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates the representation regularization-based personalization method is able to outperform other variants.
- Published
- 2021
20. FairFed: Enabling Group Fairness in Federated Learning
- Author
-
Ezzeldin, Yahya H., Yan, Shen, He, Chaoyang, Ferrara, Emilio, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Computers and Society ,Computers and Society (cs.CY) ,Machine Learning (cs.LG) - Abstract
Training ML models which are fair across different demographic groups is of critical importance due to the increased integration of ML in crucial decision-making scenarios such as healthcare and recruitment. Federated learning has been viewed as a promising solution for collaboratively training machine learning models among multiple parties while maintaining the privacy of their local data. However, federated learning also poses new challenges in mitigating the potential bias against certain populations (e.g., demographic groups), as this typically requires centralized access to the sensitive information (e.g., race, gender) of each datapoint. Motivated by the importance and challenges of group fairness in federated learning, in this work, we propose FairFed, a novel algorithm for fairness-aware aggregation to enhance group fairness in federated learning. Our proposed approach is server-side and agnostic to the applied local debiasing thus allowing for flexible use of different local debiasing methods across clients. We evaluate FairFed empirically versus common baselines for fair ML and federated learning, and demonstrate that it provides fairer models particularly under highly heterogeneous data distributions across clients. We also demonstrate the benefits of FairFed in scenarios involving naturally distributed real-life data collected from different geographical locations or departments within an organization., Accepted to appeat at AAAI 2023
- Published
- 2021
21. LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learning
- Author
-
So, Jinhyun, He, Chaoyang, Yang, Chien-Sheng, Li, Songze, Yu, Qian, Ali, Ramy E., Guler, Basak, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Cryptography and Security ,Computer Science - Distributed, Parallel, and Cluster Computing ,Statistics - Machine Learning ,Information Theory (cs.IT) ,Computer Science - Information Theory ,Machine Learning (stat.ML) ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Cryptography and Security (cs.CR) ,Machine Learning (cs.LG) - Abstract
Secure model aggregation is a key component of federated learning (FL) that aims at protecting the privacy of each user's individual model while allowing for their global aggregation. It can be applied to any aggregation-based FL approach for training a global or personalized model. Model aggregation needs to also be resilient against likely user dropouts in FL systems, making its design substantially more complex. State-of-the-art secure aggregation protocols rely on secret sharing of the random-seeds used for mask generations at the users to enable the reconstruction and cancellation of those belonging to the dropped users. The complexity of such approaches, however, grows substantially with the number of dropped users. We propose a new approach, named LightSecAgg, to overcome this bottleneck by changing the design from "random-seed reconstruction of the dropped users" to "one-shot aggregate-mask reconstruction of the active users via mask encoding/decoding". We show that LightSecAgg achieves the same privacy and dropout-resiliency guarantees as the state-of-the-art protocols while significantly reducing the overhead for resiliency against dropped users. We also demonstrate that, unlike existing schemes, LightSecAgg can be applied to secure aggregation in the asynchronous FL setting. Furthermore, we provide a modular system design and optimized on-device parallelization for scalable implementation, by enabling computational overlapping between model training and on-device encoding, as well as improving the speed of concurrent receiving and sending of chunked masks. We evaluate LightSecAgg via extensive experiments for training diverse models on various datasets in a realistic FL system with large number of users and demonstrate that LightSecAgg significantly reduces the total training time., This paper is accepted to the 5th MLSys Conference, Santa Clara, CA, USA, 2022
- Published
- 2021
22. Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training.
- Author
-
Elkordy, Ahmed Roushdy, Prakash, Saurav, and Avestimehr, Salman
- Subjects
BASIL ,GROUP rings ,DATA distribution ,PEER-to-peer architecture (Computer networks) ,INFORMATION sharing - Abstract
Decentralized (i.e., serverless) training across edge nodes can suffer substantially from potential Byzantine nodes that can degrade the training performance. However, detection and mitigation of Byzantine behaviors in a decentralized learning setting is a daunting task, especially when the data distribution at the users is heterogeneous. As our main contribution, we propose Basil, a fast and computationally efficient Byzantine-robust algorithm for decentralized training systems, which leverages a novel sequential, memory-assisted and performance-based criteria for training over a logical ring while filtering the Byzantine users. In the IID dataset setting, we provide the theoretical convergence guarantees of Basil, demonstrating its linear convergence rate. Furthermore, for the IID setting, we experimentally demonstrate that Basil is robust to various Byzantine attacks, including the strong Hidden attack, while providing up to absolute ~16% higher test accuracy over the state-of-the-art Byzantine-resilient decentralized learning approach. Additionally, we generalize Basil to the non-IID setting by proposing Anonymous Cyclic Data Sharing (ACDS), a technique that allows each node to anonymously share a random fraction of its local non-sensitive dataset (e.g., landmarks images) with all other nodes. Finally, to reduce the overall latency of Basil resulting from its sequential implementation over the logical ring, we propose Basil+ that enables Byzantine-robust parallel training across groups of logical rings, and at the same time, it retains the performance gains of Basil due to sequential training within each group. Furthermore, we experimentally demonstrate the scalability gains of Basil+ through different sets of experiments. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF
23. FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
- Author
-
He, Chaoyang, Balasubramanian, Keshav, Ceyani, Emir, Yang, Carl, Xie, Han, Sun, Lichao, He, Lifang, Yang, Liangwei, Yu, Philip S., Rong, Yu, Zhao, Peilin, Huang, Junzhou, Annavaram, Murali, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Artificial Intelligence (cs.AI) ,Computer Science - Distributed, Parallel, and Cluster Computing ,Computer Science - Artificial Intelligence ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
Graph Neural Network (GNN) research is rapidly growing thanks to the capacity of GNNs in learning distributed representations from graph-structured data. However, centralizing a massive amount of real-world graph data for GNN training is prohibitive due to privacy concerns, regulation restrictions, and commercial competitions. Federated learning (FL), a trending distributed learning paradigm, provides possibilities to solve this challenge while preserving data privacy. Despite recent advances in vision and language domains, there is no suitable platform for the FL of GNNs. To this end, we introduce FedGraphNN, an open FL benchmark system that can facilitate research on federated GNNs. FedGraphNN is built on a unified formulation of graph FL and contains a wide range of datasets from different domains, popular GNN models, and FL algorithms, with secure and efficient system support. Particularly for the datasets, we collect, preprocess, and partition 36 datasets from 7 domains, including both publicly available ones and specifically obtained ones such as hERG and Tencent. Our empirical analysis showcases the utility of our benchmark system, while exposing significant challenges in graph FL: federated GNNs perform worse in most datasets with a non-IID split than centralized GNNs; the GNN model that attains the best result in the centralized setting may not maintain its advantage in the FL setting. These results imply that more research efforts are needed to unravel the mystery behind federated GNNs. Moreover, our system performance analysis demonstrates that the FedGraphNN system is computationally efficient and secure to large-scale graphs datasets. We maintain the source code at https://github.com/FedML-AI/FedGraphNN., Our shorter versions are accepted to ICLR 2021 Workshop on Distributed and Private Machine Learning(DPML) and MLSys 2021 GNNSys Workshop on Graph Neural Networks and Systems. The full version is under review
- Published
- 2021
24. PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
- Author
-
He, Chaoyang, Li, Shen, Soltanolkotabi, Mahdi, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Machine Learning (cs.LG) - Abstract
The size of Transformer models is growing at an unprecedented pace. It has only taken less than one year to reach trillion-level parameters after the release of GPT-3 (175B). Training such models requires both substantial engineering efforts and enormous computing resources, which are luxuries most research teams cannot afford. In this paper, we propose PipeTransformer, which leverages automated and elastic pipelining and data parallelism for efficient distributed training of Transformer models. PipeTransformer automatically adjusts the pipelining and data parallelism by identifying and freezing some layers during the training, and instead allocates resources for training of the remaining active layers. More specifically, PipeTransformer dynamically excludes converged layers from the pipeline, packs active layers into fewer GPUs, and forks more replicas to increase data-parallel width. We evaluate PipeTransformer using Vision Transformer (ViT) on ImageNet and BERT on GLUE and SQuAD datasets. Our results show that PipeTransformer attains a 2.4 fold speedup compared to the state-of-the-art baseline. We also provide various performance analyses for a more comprehensive understanding of our algorithmic and system-wise design. We also develop open-sourced flexible APIs for PipeTransformer, which offer a clean separation among the freeze algorithm, model definitions, and training accelerations, hence allowing it to be applied to other algorithms that require similar freezing strategies.
- Published
- 2021
25. Privacy in Retrieval, Computing, and Learning.
- Author
-
Ulukus, Sennur, Avestimehr, Salman, Gastpar, Michael, Jafar, Syed, Tandon, Ravi, and Tian, Chao
- Subjects
CODING theory ,PRIVACY ,DATA privacy ,DISTRIBUTED computing ,INFORMATION theory ,GRID computing - Abstract
The increasing prevalence of massive datasets makes the outsourcing of storage and computation tasks to distributed servers a necessity. This raises a number of concerns regarding the security and integrity of stored information, the privacy of accessing desired information, the communication overhead of distributed systems, the latency, reliability, and complexity of distributed computing, and privacy in distributed training and learning systems. Recent breakthroughs from coding, communication, and information-theoretic perspectives have opened up exciting new research avenues for these topics. There are many theoretical and practical open problems. This Special Issue is dedicated to communication theory, coding theory, information theory, signal processing, and networking aspects of privacy in information retrieval, privacy in coded computing over distributed servers, and privacy in distributed learning. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF
26. Private Retrieval, Computing, and Learning: Recent Progress and Future Challenges.
- Author
-
Ulukus, Sennur, Avestimehr, Salman, Gastpar, Michael, Jafar, Syed A., Tandon, Ravi, and Tian, Chao
- Subjects
INTERNET privacy ,DISTRIBUTED computing ,DATA privacy ,CYBERSPACE ,INFORMATION retrieval ,GRID computing ,PARALLEL processing - Abstract
Most of our lives are conducted in the cyberspace. The human notion of privacy translates into a cyber notion of privacy on many functions that take place in the cyberspace. This article focuses on three such functions: how to privately retrieve information from cyberspace (privacy in information retrieval), how to privately leverage large-scale distributed/parallel processing (privacy in distributed computing), and how to learn/train machine learning models from private data spread across multiple users (privacy in distributed (federated) learning). The article motivates each privacy setting, describes the problem formulation, summarizes breakthrough results in the history of each problem, and gives recent results and discusses some of the major ideas that emerged in each field. In addition, the cross-cutting techniques and interconnections between the three topics are discussed along with a set of open problems and challenges. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF
27. Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
- Author
-
He, Chaoyang, Annavaram, Murali, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Vision and Pattern Recognition (cs.CV) ,Computer Science - Computer Vision and Pattern Recognition ,Machine Learning (cs.LG) - Abstract
Scaling up the convolutional neural network (CNN) size (e.g., width, depth, etc.) is known to effectively improve model accuracy. However, the large model size impedes training on resource-constrained edge devices. For instance, federated learning (FL) may place undue burden on the compute capability of edge nodes, even though there is a strong practical need for FL due to its privacy and confidentiality properties. To address the resource-constrained reality of edge devices, we reformulate FL as a group knowledge transfer training algorithm, called FedGKT. FedGKT designs a variant of the alternating minimization approach to train small CNNs on edge nodes and periodically transfer their knowledge by knowledge distillation to a large server-side CNN. FedGKT consolidates several advantages into a single framework: reduced demand for edge computation, lower communication bandwidth for large CNNs, and asynchronous training, all while maintaining model accuracy comparable to FedAvg. We train CNNs designed based on ResNet-56 and ResNet-110 using three distinct datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D. variants. Our results show that FedGKT can obtain comparable or even slightly higher accuracy than FedAvg. More importantly, FedGKT makes edge training affordable. Compared to the edge training using FedAvg, FedGKT demands 9 to 17 times less computational power (FLOPs) on edge devices and requires 54 to 105 times fewer parameters in the edge CNN. Our source code is released at FedML (https://fedml.ai)., This paper is accepted to NeurIPS 2020. We propose FedGKT, attempting to address one of the core problems of federated learning: training deep neural networks in resource-constrained edge devices
- Published
- 2020
28. FedML: A Research Library and Benchmark for Federated Machine Learning
- Author
-
He, Chaoyang, Li, Songze, So, Jinhyun, Zeng, Xiao, Zhang, Mi, Wang, Hongyi, Wang, Xiaoyang, Vepakomma, Praneeth, Singh, Abhishek, Qiu, Hang, Zhu, Xinghua, Wang, Jianzong, Shen, Li, Zhao, Peilin, Kang, Yan, Liu, Yang, Raskar, Ramesh, Yang, Qiang, Annavaram, Murali, and Avestimehr, Salman
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Statistics - Machine Learning ,Machine Learning (stat.ML) ,Machine Learning (cs.LG) - Abstract
Federated learning (FL) is a rapidly growing research field in machine learning. However, existing FL libraries cannot adequately support diverse algorithmic development; inconsistent dataset and model usage make fair algorithm comparison challenging. In this work, we introduce FedML, an open research library and benchmark to facilitate FL algorithm development and fair performance comparison. FedML supports three computing paradigms: on-device training for edge devices, distributed computing, and single-machine simulation. FedML also promotes diverse algorithmic research with flexible and generic API design and comprehensive reference baseline implementations (optimizer, models, and datasets). We hope FedML could provide an efficient and reproducible means for developing and evaluating FL algorithms that would benefit the FL research community. We maintain the source code, documents, and user community at https://fedml.ai., This is FedML white paper V3. Homepage: https://fedml.ai; GitHub: https://github.com/FedML-AI/FedML; In V3, More advanced algorithms and IoT device training are supported, please check here: https://github.com/FedML-AI/FedML/blob/master/fedml_iot/
- Published
- 2020
29. Collage Inference: Achieving low tail latency during distributed image classification using coded redundancy models
- Author
-
Narra, Krishna, Lin, Zhifeng, Ananthanarayanan, Ganesh, Avestimehr, Salman, and Annavaram, Murali
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Distributed, Parallel, and Cluster Computing ,Statistics - Machine Learning ,Machine Learning (stat.ML) ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
Reducing the latency variance in machine learning inference is a key requirement in many applications. Variance is harder to control in a cloud deployment in the presence of stragglers. In spite of this challenge, inference is increasingly being done in the cloud, due to the advent of affordable machine learning as a service (MLaaS) platforms. Existing approaches to reduce variance rely on replication which is expensive and partially negates the affordability of MLaaS. In this work, we argue that MLaaS platforms also provide unique opportunities to cut the cost of redundancy. In MLaaS platforms, multiple inference requests are concurrently received by a load balancer which can then create a more cost-efficient redundancy coding across a larger collection of images. We propose a novel convolutional neural network model, Collage-CNN, to provide a low-cost redundancy framework. A Collage-CNN model takes a collage formed by combining multiple images and performs multi-image classification in one shot, albeit at slightly lower accuracy. We then augment a collection of traditional single image classifiers with a single Collage-CNN classifier which acts as a low-cost redundant backup. Collage-CNN then provides backup classification results if a single image classification straggles. Deploying the Collage-CNN models in the cloud, we demonstrate that the 99th percentile tail latency of inference can be reduced by 1.47X compared to replication based approaches while providing high accuracy. Also, variation in inference latency can be reduced by 9X with a slight increase in average inference latency., 4 pages, CodML workshop at International Conference on Machine Learning (ICML 2019). arXiv admin note: text overlap with arXiv:1904.12222
- Published
- 2019
30. Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
- Author
-
Li, Youjie, Yu, Mingchao, Li, Songze, Avestimehr, Salman, Kim, Nam Sung, and Schwing, Alexander
- Subjects
FOS: Computer and information sciences ,Computer Science - Machine Learning ,Computer Science - Distributed, Parallel, and Cluster Computing ,Statistics - Machine Learning ,Machine Learning (stat.ML) ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Machine Learning (cs.LG) - Abstract
Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4x compared to conventional approaches., Accepted at NeurIPS 2018
- Published
- 2018
31. Improved Sparse Recovery Thresholds with Two-Step Reweighted $\ell_1$ Minimization
- Author
-
Khajehnejad, M. Amin, Xu, Weiyu, Avestimehr, Salman, and Hassibi, Babak
- Subjects
FOS: Computer and information sciences ,Statistics::Machine Learning ,Information Theory (cs.IT) ,Computer Science - Information Theory - Abstract
It is well known that $\ell_1$ minimization can be used to recover sufficiently sparse unknown signals from compressed linear measurements. In fact, exact thresholds on the sparsity, as a function of the ratio between the system dimensions, so that with high probability almost all sparse signals can be recovered from iid Gaussian measurements, have been computed and are referred to as "weak thresholds" \cite{D}. In this paper, we introduce a reweighted $\ell_1$ recovery algorithm composed of two steps: a standard $\ell_1$ minimization step to identify a set of entries where the signal is likely to reside, and a weighted $\ell_1$ minimization step where entries outside this set are penalized. For signals where the non-sparse component has iid Gaussian entries, we prove a "strict" improvement in the weak recovery threshold. Simulations suggest that the improvement can be quite impressive-over 20% in the example we consider., accepted in ISIT 2010
- Published
- 2010
32. Weighted $\ell_1$ Minimization for Sparse Recovery with Prior Information
- Author
-
Khajehnejad, M. Amin, Xu, Weiyu, Avestimehr, Salman, and Hassibi, Babak
- Subjects
FOS: Computer and information sciences ,Information Theory (cs.IT) ,Computer Science - Information Theory - Abstract
In this paper we study the compressed sensing problem of recovering a sparse signal from a system of underdetermined linear equations when we have prior information about the probability of each entry of the unknown signal being nonzero. In particular, we focus on a model where the entries of the unknown vector fall into two sets, each with a different probability of being nonzero. We propose a weighted $\ell_1$ minimization recovery algorithm and analyze its performance using a Grassman angle approach. We compute explicitly the relationship between the system parameters (the weights, the number of measurements, the size of the two sets, the probabilities of being non-zero) so that an iid random Gaussian measurement matrix along with weighted $\ell_1$ minimization recovers almost all such sparse signals with overwhelming probability as the problem dimension increases. This allows us to compute the optimal weights. We also provide simulations to demonstrate the advantages of the method over conventional $\ell_1$ optimization., 5 Pages, Submitted to ISIT 2009
- Published
- 2009
33. On Source-Channel Separation in Networks
- Author
-
Avestimehr, Salman, Caire, Giuseppe, and Tse, David
- Subjects
FOS: Computer and information sciences ,Information Theory (cs.IT) ,Computer Science - Information Theory - Abstract
This paper has been withdrawn., Comment: This paper has been withdrawn
- Published
- 2009
34. Outage Capacity of the Fading Relay Channel in the Low SNR Regime
- Author
-
Avestimehr, Salman and Tse, David N. C.
- Subjects
FOS: Computer and information sciences ,Computer Science::Performance ,Information Theory (cs.IT) ,Computer Science - Information Theory ,ComputerSystemsOrganization_COMPUTER-COMMUNICATIONNETWORKS ,Computer Science::Networking and Internet Architecture ,Data_CODINGANDINFORMATIONTHEORY ,Computer Science::Information Theory - Abstract
In slow fading scenarios, cooperation between nodes can increase the amount of diversity for communication. We study the performance limit in such scenarios by analyzing the outage capacity of slow fading relay channels. Our focus is on the low SNR and low outage probability regime, where the adverse impact of fading is greatest but so are the potential gains from cooperation. We showed that while the standard Amplify-Forward protocol performs very poorly in this regime, a modified version we called the Bursty Amplify-Forward protocol is optimal and achieves the outage capacity of the network. Moreover, this performance can be achieved without a priori channel knowledge at the receivers. In contrast, the Decode-Forward protocol is strictly sub-optimal in this regime. Our results directly yield the outage capacity per unit energy of fading relay channels.
- Published
- 2006
35. MISO Broadcast Channel With Hybrid CSIT: Beyond Two Users.
- Author
-
Lashgari, Sina, Tandon, Ravi, and Avestimehr, Salman
- Subjects
DEGREES of freedom ,ANALYSIS of variance ,PHASE equilibrium ,MECHANICS (Physics) ,MISO - Abstract
We study the impact of heterogeneity of channel-state-information available at the transmitters (CSIT) on the capacity of broadcast channels with a multiple-antenna transmitter and $k$ single-antenna receivers (MISO BC). In particular, we consider the $k$ -user MISO BC, where the CSIT with respect to each receiver can be either instantaneous/perfect, delayed, or not available; and we study the impact of this heterogeneity of CSIT on the degrees-of-freedom (DoFs) of such network. We first focus on the three-user MISO BC, and we completely characterize the DoF region for all possible heterogeneous CSIT configurations, assuming linear encoding strategies at the transmitters. The result shows that the state-of-the-art achievable schemes in the literature are indeed sum-DoF optimal, when restricted to linear encoding schemes. To prove the result, we develop a novel bound, called interference decomposition bound, which provides a lower bound on the interference dimension at a receiver which supplies delayed CSIT based on the average dimension of constituents of that interference, thereby decomposing the interference into its individual components. Furthermore, we extend our outer bound on the DoF region to the general $k$ -user MISO BC, and demonstrate that it leads to an approximate characterization of linear sum-DoF to within an additive gap of 0.5 for a broad range of CSIT configurations. Moreover, for the special case where only one receiver supplies delayed CSIT, we completely characterize the linear sum-DoF. [ABSTRACT FROM PUBLISHER]
- Published
- 2016
- Full Text
- View/download PDF
36. Asymptotic justification of bandlimited interpolation of graph signals for semi-supervised learning.
- Author
-
Anis, Aamir, El Gamal, Aly, Avestimehr, Salman, and Ortega, Antonio
- Published
- 2015
- Full Text
- View/download PDF
37. Three-user MISO broadcast channel: How much can CSIT heterogeneity help?
- Author
-
Lashgari, Sina, Tandon, Ravi, and Avestimehr, Salman
- Published
- 2015
- Full Text
- View/download PDF
38. A latent social approach to YouTube popularity prediction.
- Author
-
Nwana, Amandianeze O, Avestimehr, Salman, and Tsuhan Chen
- Published
- 2013
- Full Text
- View/download PDF
39. Federated Learning of Generative Image Priors for MRI Reconstruction.
- Author
-
Elmas G, Dar SUH, Korkmaz Y, Ceyani E, Susam B, Ozbey M, Avestimehr S, and Cukur T
- Subjects
- Humans, Magnetic Resonance Imaging methods, Image Processing, Computer-Assisted methods, Deep Learning
- Abstract
Multi-institutional efforts can facilitate training of deep MRI reconstruction models, albeit privacy risks arise during cross-site sharing of imaging data. Federated learning (FL) has recently been introduced to address privacy concerns by enabling distributed training without transfer of imaging data. Existing FL methods employ conditional reconstruction models to map from undersampled to fully-sampled acquisitions via explicit knowledge of the accelerated imaging operator. Since conditional models generalize poorly across different acceleration rates or sampling densities, imaging operators must be fixed between training and testing, and they are typically matched across sites. To improve patient privacy, performance and flexibility in multi-site collaborations, here we introduce Federated learning of Generative IMage Priors (FedGIMP) for MRI reconstruction. FedGIMP leverages a two-stage approach: cross-site learning of a generative MRI prior, and prior adaptation following injection of the imaging operator. The global MRI prior is learned via an unconditional adversarial model that synthesizes high-quality MR images based on latent variables. A novel mapper subnetwork produces site-specific latents to maintain specificity in the prior. During inference, the prior is first combined with subject-specific imaging operators to enable reconstruction, and it is then adapted to individual cross-sections by minimizing a data-consistency loss. Comprehensive experiments on multi-institutional datasets clearly demonstrate enhanced performance of FedGIMP against both centralized and FL methods based on conditional models.
- Published
- 2023
- Full Text
- View/download PDF
Catalog
Discovery Service for Jio Institute Digital Library
For full access to our library's resources, please sign in.