The fault diagnosis of a marine turbocharger system is very crucial for realizing intelligent operation and maintenance in a big data analysis context. In order to improve the diagnostic rate of faults in engineering applications, in this paper, a new unsupervised machine learning algorithm, which is based on one-class support vector machine (OSVM), affinity propagation (AP) and Gaussian mixture model (GMM), called OAGFD is proposed for fault diagnosis. OSVM was firstly used to divide samples of marine turbocharger system into normal and fault samples, and only the fault samples are used in following steps to identify specific fault types. The AP was adopted automatically to provide an initial value for expectation maximization, which can obtain the maximum value of iteration parameters. The GMM is used to classify faults of marine turbocharger system and output the fault diagnosis results. Finally, the OAGFD is validated by actual data. The experiment results show that OAGFD can quickly and accurately be trained. The OAGFD method can achieve higher identification accuracy for multi-faults of marine turbocharger system and takes on faster operation speed and stronger generalization ability than tradition methods. It is an efficient and unsupervised fault diagnosis technique and has both theoretical and practical value. This research provides a new method for automatic fault diagnosis of the marine turbocharger system.