Search

We consider synchronous data-parallel neural network training with a fixed large batch size. While the large batch size provides a high degree of parallelism, it degrades the generalization performance due to the low gradient noise scale. We propose a general learning rate adjustment framework and three critical heuristics that tackle the poor generalization issue. The key idea is to adjust the learning rate based on geometric information of loss landscape and encourage the model to converge into a flat minimum that is known to better generalize to the unknown data. Our empirical study demonstrates that the Hessian-aware learning rate schedule remarkably improves the generalization performance in large-batch training. For CIFAR-10 classification with ResNet20, our method achieves 92.31% accuracy using 16,384 batch size, which is close to 92.83% achieved using 128 batch size, at a negligible extra computational cost. • Hessian information allows to properly adjust noise scale in large-batch training. • Too early learning rate decay harms underlying margin distribution. • The minimum learning rate after the decay strongly affects the model sharpness. • The length of noise scale transition affects the final generalization performance. [ABSTRACT FROM AUTHOR]

We study the impact of heterogeneity of channel-state-information available at the transmitters (CSIT) on the capacity of broadcast channels with a multiple-antenna transmitter and $k$ single-antenna receivers (MISO BC). In particular, we consider the $k$ -user MISO BC, where the CSIT with respect to each receiver can be either instantaneous/perfect, delayed, or not available; and we study the impact of this heterogeneity of CSIT on the degrees-of-freedom (DoFs) of such network. We first focus on the three-user MISO BC, and we completely characterize the DoF region for all possible heterogeneous CSIT configurations, assuming linear encoding strategies at the transmitters. The result shows that the state-of-the-art achievable schemes in the literature are indeed sum-DoF optimal, when restricted to linear encoding schemes. To prove the result, we develop a novel bound, called interference decomposition bound, which provides a lower bound on the interference dimension at a receiver which supplies delayed CSIT based on the average dimension of constituents of that interference, thereby decomposing the interference into its individual components. Furthermore, we extend our outer bound on the DoF region to the general $k$ -user MISO BC, and demonstrate that it leads to an approximate characterization of linear sum-DoF to within an additive gap of 0.5 for a broad range of CSIT configurations. Moreover, for the special case where only one receiver supplies delayed CSIT, we completely characterize the linear sum-DoF. [ABSTRACT FROM PUBLISHER]

Curation of large, diverse MRI datasets via multi-institutional collaborations can help improve learning of generalizable synthesis models that reliably translate source- onto target-contrast images. To facilitate collaborations, federated learning (FL) adopts decentralized model training while mitigating privacy concerns by avoiding sharing of imaging data. However, conventional FL methods can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident within and across imaging sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) that improves reliability against data heterogeneity via model specialization to individual sites and synthesis tasks (i.e., source-target contrasts). To do this, pFLSynth leverages an adversarial model equipped with novel personalization blocks that control the statistics of generated feature maps across the spatial/channel dimensions, given latent variables specific to sites and tasks. To further promote communication efficiency and site specialization, partial network aggregation is employed over later generator stages while earlier generator stages and the discriminator are trained locally. As such, pFLSynth enables multi-task training of multi-site synthesis models with high generalization performance across sites and tasks. Comprehensive experiments demonstrate the superior performance and reliability of pFLSynth in MRI synthesis against prior federated methods. [Display omitted] • A novel personalized federated learning method for multi-contrast MRI synthesis. • A novel generator equipped with personalization blocks to improve model specialization. • Partial network aggregation to improve communication efficiency and personalization. • State-of-the-art performance in MRI synthesis for common and variable tasks across sites. [ABSTRACT FROM AUTHOR]

Local Stochastic Gradient Descent (SGD) with periodic model averaging (FedAvg) is a foundational algorithm in Federated Learning. The algorithm independently runs SGD on multiple clients and periodically averages the model across all the clients. This periodic model averaging potentially causes a significant model discrepancy across the clients making the global loss converge slowly. While recent advanced optimization methods tackle the issue focused on non-IID settings, there still exists the model discrepancy issue due to the underlying periodic model averaging. We propose a partial model averaging framework that mitigates the model discrepancy issue in Federated Learning. The partial averaging encourages the local models to stay close to each other on parameter space, and it enables to more effectively minimize the global loss. We extensively evaluate the performance of the partial averaging strategy using CIFAR-10/100 and FEMNIST benchmarks. Given a fixed number of training iterations and a large number of clients (128), the partial averaging achieves up to 2.2% higher accuracy than the periodic full averaging. • A novel partial model averaging scheme that accelerates the federated optimization. • Frequent partial model synchronizations strongly suppress the model discrepancy across clients. • While maintaining the total data transfer size, the partial averaging accelerate the convergence of global loss. [ABSTRACT FROM AUTHOR]