• Reviews more than 45 recent solutions papers, and more than 40 different privacy-preserving deep learning techniques. • Proposes a multi-level taxonomy that classifies the privacy-preserving deep learning techniques. • Summarizes evaluation results of the reviewed solutions with respect to performance metrics. • Discusses and outline a number of learned lessons of each privacy-preserving task. • Presents solutions comparison, highlights open research challenges and provides some recommendations for future research. Deep learning is one of the advanced approaches of machine learning, and has attracted a growing attention in the recent years. It is used nowadays in different domains and applications such as pattern recognition, medical prediction, and speech recognition. Differently from traditional learning algorithms, deep learning can overcome the dependency on hand-designed features. Deep learning experience is particularly improved by leveraging powerful infrastructures such as clouds and adopting collaborative learning for model training. However, this comes at the expense of privacy, especially when sensitive data are processed during the training and the prediction phases, as well as when training model is shared. In this paper, we provide a review of the existing privacy-preserving deep learning techniques, and propose a novel multi-level taxonomy, which categorizes the current state-of-the-art privacy-preserving deep learning techniques on the basis of privacy-preserving tasks at the top level, and key technological concepts at the base level. This survey further summarizes evaluation results of the reviewed solutions with respect to defined performance metrics. In addition, it derives a set of learned lessons from each privacy-preserving task. Finally, it highlights open research challenges and provides some recommendations as future research directions. [ABSTRACT FROM AUTHOR]