Search

We present NetReduce, a novel RDMA-compatible in-network reduction architecture to accelerate distributed DNN training. Compared to existing designs, NetReduce maintains a reliable connection between end-hosts in the Ethernet and does not terminate the connection in the network. The advantage of doing so is that we can fully reuse the designs of congestion control and reliability in RoCE. In the meanwhile, we do not need to implement a high-cost network protocol processing stack in the switch, as IB does. The prototype implemented by using FPGA is an out-of-box solution without modifying commodity devices such as NICs or switches. For the coordination between the end-host and the switch, NetReduce customizes the transport protocol only on the first packet in a data message to comply with RoCE v2. The special status monitoring module is designed to reuse the reliability mechanism of RoCE v2 for dealing with packet loss. A message-level credit-based flow control algorithm is also proposed to fully utilize bandwidth and avoid buffer overflow. We study the effects of intra bandwidth on the training performance in multi-machines multi-GPUs scenario and give sufficient conditions for hierarchical NetReduce to outperform other algorithms. We also extend the design from rack-level aggregation to more general spine-leaf topology in the data center. NetReduce accelerates the training up to 1.7x and 1.5x for CNN-based CV and transformer-based NLP tasks, respectively. Simulations on large-scale systems indicate the superior scalability of NetReduce to the state-of-the-art ring all-reduce.

This paper considers the Byzantine fault-tolerance problem in distributed stochastic gradient descent (D-SGD) method - a popular algorithm for distributed multi-agent machine learning. In this problem, each agent samples data points independently from a certain data-generating distribution. In the fault-free case, the D-SGD method allows all the agents to learn a mathematical model best fitting the data collectively sampled by all agents. We consider the case when a fraction of agents may be Byzantine faulty. Such faulty agents may not follow a prescribed algorithm correctly, and may render traditional D-SGD method ineffective by sharing arbitrary incorrect stochastic gradients. We propose a norm-based gradient-filter, named comparative gradient elimination (CGE), that robustifies the D-SGD method against Byzantine agents. We show that the CGE gradient-filter guarantees fault-tolerance against a bounded fraction of Byzantine agents under standard stochastic assumptions, and is computationally simpler compared to many existing gradient-filters such as multi-KRUM, geometric median-of-means, and the spectral filters. We empirically show, by simulating distributed learning on neural networks, that the fault-tolerance of CGE is comparable to that of existing gradient-filters. We also empirically show that exponential averaging of stochastic gradients improves the fault-tolerance of a generic gradient-filter., Comment: The report includes 52 pages, and 16 figures. Extension of our prior work on Byzantine fault-tolerant distribution optimization (arXiv:1903.08752 and doi:10.1145/3382734.3405748) to Byzantine fault-tolerant distributed machine learning; Updated to the full version of workshop paper in DSN-DSML '21

Many empirical networks are intrinsically pluralistic, with interactions occurring within groups of arbitrary agents. Then the agent in the network can be influenced by types of neighbors, common examples include similarity, opposition, and negligibility. Although the influence of neighbors can be described as an amicable and antagonistic relationship in complex real-world systems accurately, and the research on the dynamic process of public opinion evolution with different types of influence is valuable, few studies have mentioned that issue. In this paper, we develop a novel model on networks of agents with the bi-directional bounded thresholds for studying the evolution of opinion dynamics. We define the scope of individual assimilation and exclusion to identify different types of neighbors and calculate the impact of the corresponding neighbors on the individuals by converting the opinion difference. The simulation results show that the proposed mechanism can effectively explain the formation of bi-polarization during opinion evolution and the settings of the bi-directional bounded thresholds significantly influence the eventual distribution of opinions. Furthermore, we explore the impacts of the initial conditions and the structure of the small-world network on the evolution of opinions.

The COVID-19 outbreak was announced as a global pandemic by the World Health Organisation in March 2020 and has affected a growing number of people in the past few weeks. In this context, advanced artificial intelligence techniques are brought to the fore in responding to fight against and reduce the impact of this global health crisis. In this study, we focus on developing some potential use-cases of intelligent speech analysis for COVID-19 diagnosed patients. In particular, by analysing speech recordings from these patients, we construct audio-only-based models to automatically categorise the health state of patients from four aspects, including the severity of illness, sleep quality, fatigue, and anxiety. For this purpose, two established acoustic feature sets and support vector machines are utilised. Our experiments show that an average accuracy of .69 obtained estimating the severity of illness, which is derived from the number of days in hospitalisation. We hope that this study can foster an extremely fast, low-cost, and convenient way to automatically detect the COVID-19 disease.

Long ground-Rydberg coherence lifetime is interesting for implementing high-fidelity quantum logic gates, many-body physics, and other quantum information protocols. However, the potential formed by a conventional far-off-resonance red-detuned optical-dipole trap (ODT) is usually repulsive for Rydberg atoms, which will result in fast atom loss and low repetition rate of the experimental sequence. These issues can be addressed by a magic ODT. We performed the calculation of ODT's magic detuning for confinement of cesium ground state and Rydberg state with the same potential well. We used a sum-over-states method to calculate the dynamic polarizabilities of $6S_{1/2}$ ground state and highly-excited ($nS_{1/2}$ and $nP_{3/2}$) Rydberg state of cesium atoms, and identify corresponding magic detuning for optical wavelengths in the range of $850 - 2000$ nm. We estimated the trapping lifetime of cesium Rydberg atoms confined in the magic ODT by including different dissipative mechanisms. Furthermore, we have experimentally realized an 1879.43-nm single-frequency laser system with a watt-level output power for setting up the magic ODT for $6S_{1/2}$ ground-state and $84P_{3/2}$ Rydberg-state cesium cold atoms., Comment: 18 pages, 5 figures

The rotational bands in the neutron-rich nuclei $^{153-157}$Pm are investigated by a particle-number conserving method. The kinematic moments of inertia for the 1-quasiparticle bands in odd-$A$ Pm isotopes $^{153, 155, 157}$Pm are reproduced quite well by the present calculation. By comparison between the experimental and calculated moments of inertia for the three 2-quasiparticle bands in the odd-odd nuclei $^{154, 156}$Pm, their configurations and bandhead spins have been assigned properly. For the 2-quasiparticle band in $^{154}$Pm, the configuration is assigned as $\pi5/2^-[532]\otimes \nu3/2^-[521]$ ($K^\pi=4^+$) with the bandhead spin $I_0=4\hbar$. In $^{156}$Pm, the same configuration and bandhead spin assignments have been made for the 2-quasiparticle band with lower excitation energy. The configuration $\pi5/2^+[413]\otimes \nu5/2^+[642]$ ($K^\pi=5^+$) with the bandhead spin $I_0=5\hbar$ is assigned for that with higher excitation energy., Comment: 6 figures, 2 tables