Search

With the onset of the big data era, data has become an unprecedented valuable resource of intelligence and brings new opportunities to governments, companies, and research institutions to gain deep insight into their own problems and develop better solutions. Artificial intelligence methods, such as machine learning and deep learning algorithms, which are able to analyze such enormous volumes of data, are the crucial techniques to the big data industry. However, due to the nature of big data, which is typically generated by multiple parties and stored in a geographically distributed manner, it is impossible to process it in a centralized computing center. Therefore, designing efficient and effective multi-party collaborative learning is in dire need. Unlike most manually designed multi-party collaborative learning methods, in this work, motivated by learning to learn, we are exploring in utilizing deep meta-learning to learn learning systems over multiple computing nodes with their own independent datasets.We first propose a meta-learning approach to coordinate the learning process in the master-slave type of distributed systems. Specifically, we utilize a recurrent neural network (RNN) in the parameter server (the master) to learn to aggregate the gradients from the workers (the slaves). We design a coordinatewise preprocessing and postprocessing method to make the neural-network-based aggregator more robust. Besides, to address the fault tolerance, especially the Byzantine attack, in distributed machine learning systems, we propose an RNN aggregator with additional loss information (ARNN) to improve system resilience. We conduct extensive experiments to demonstrate the effectiveness of the RNN aggregator, and also show that it can be easily generalized and achieve remarkable performance when transferred to other distributed systems. Moreover, under majoritarian Byzantine attacks, the ARNN aggregator outperforms the Krum, the state-of-art fault tolerance aggregation method. In addition, our RNN aggregator enables the server to aggregate gradients from variant local models, which significantly improve the scalability of distributed learning.In addition, we extend the work of meta-aggregator in distributed learning and propose meta-learning approaches to coordinate more general multi-party collaborative learning. Specifically, we design learning to learn distributed learning and learning to learn decentralized learning. In distributed learning, we introduce meta-encoders and meta-decoders to workers and equip the central server with meta-aggregator. Meanwhile, we only apply meta-encoders and meta-decoders to workers in decentralized learning. Specifically, the meta-encoders and meta-decoders make the computing nodes inside the network to exchange information by fixed-sized compressed hidden gradient embeddings, which significantly reduce the communication overhead in the network and also enable multi-party heterogeneous learning. The entire meta-learning controlling system enables the geographically distributed multiple computing nodes to cooperate together and learn the centralized and decentralized parallel optimization methods for distributed learning and decentralized learning, respectively, which accelerate the collaboratively training process. We conduct extensive experiments to demonstrate the effectiveness of learning to learn distributed learning and learning to learn decentralized learning. We also show that they can be easily generalized and achieve remarkable performance when transferred to other multi-party systems. Moreover, the meta-learning based multi-party learning significantly reduces the averaging transmission cost when compared to the state-of-the-art parallel stochastic gradient descent algorithm. Besides, learning to learn enabled multi-party heterogeneous learning also improves the scalability of traditional distributed learning and decentralized learning.