Search

The fluctuation of stock price is a major concern for stock investors. In order to provide investors with decision-making suggestions for buying or selling stocks, we use big data computing platform-Spark to perform binary classification prediction of stocks’ rise and fall according to the price and related information in stock trading day, and compare four classification algorithms: naive bayes, random forest, decision tree, and logistic regression from four metrics: run time, classification accuracy, AUC(Area under the receiver operating characteristic curve) and PR(Area under the Precision-Recall curve). Experimental results show that: (1) the run time is negatively correlated with the scale of the cluster. When the number of node in the cluster is increased by 3, the run time is reduced to about two-fifths; 2. Naive Bayes has the lowest accuracy 0.7436, and the accuracy of random forests, decision trees, and logistic regression are 0.8297, 0.8318, 0.8085 respectively; (3) both the random forest and decision tree algorithm have a higher AUC and PR value, the values of the random forest are 0.6891,0.7497 and the value of the decision tree are 0.7165,0.7485. The evaluation results show that the random forest and decision tree algorithms are more suitable for stock classification prediction, because they pay more attention to the features with large information gain values and can reduce the error of the classification.

Nowadays the phenomenon of flight delays and cancellations is becoming more and more serious. Flight delays and cancellations not only waste transportation resources, but also affect passengers’ travel plans, which cause an increase in passenger discontent and complaint rates[2]. The passengers’ dissatisfaction and distrust of airlines seriously damage the airlines’ corporate reputation and then affect passengers’ loyalty. Therefore, based on the information of 5 million flight data in the United States in 2016, we predicted the flight cancellations based on four classification algorithms: logistics regression, support vector machine (SVM), naive bayes and decision tree. We compared the training time of these algorithms by changing the number of nodes in the Spark cluster[9], and we also compared the classification accuracy, AUC (Area Under Curve) and PR(Precision-Recall). Experimental results show that: (1) When the number of nodes was 1, the average training time of these algorithms was 460 seconds. When the number of nodes was 3,5,7,9, the corresponding average training time were 207.25 seconds, 89.75 seconds, 44 seconds, 42 seconds, and when the number of nodes was 11, the average training time was 43.5 seconds; (2) Starting from 7 nodes, as the number of nodes increased while the decline rate of the average training time slowed down; (3) When the number of nodes was 1 and 3, the training time of naive bayes was the shortest, which were 386 seconds and 185 seconds respectively. When the number of nodes was 5,7,9 and11, the training time of SVM was the shortest, which were 72 seconds, 21seconds, 19 seconds and 18 seconds respectively; (4) the classification accuracy of SVM and decision tree was almost 90%, but naive bayes was just about 50.8% and logistics regression was only about 62.4%; (5) Both the AUC and PR of the decision tree algorithm were the highest: AUC was 0.558 and PR was 0.439. Therefore, the decision tree algorithm was the most suitable for predicting whether the flight would be cancelled or not, and there was a 90% probability that the prediction would be successful.

Credit evaluation becomes more and more important in the financial field. Based on the data set of Lending Club-an american P2P platform, we apply four classification algorithms: logistic regression, naive bayes, decision tree and support vector machine, and compare their classfication accuracy, AUC(Area Under Curve)and PR(Precision-recall Rate). By changing the number of nodes of Spark cluster, we compare the run time of these algorithms and analyze the variance of the run time of each algorithm with the increase of the number of nodes. Experimental results show: (1) among these algorithms, the classification accuracy of naive bayes and decision tree is over 40% higher than that of logistic regression and support vector machine when the number of nodes is 3;(2) the AUC value of naive bayes and decision tree is over 45% higher than the other two algorithms; (3)as for the PR value, the performance of naive bayes and decision tree is better, the gap is over 40%; (4)In pseudo-distribution state, the run time of naive bayes is the shortest, but the run time of decision tree decreases obviously from 109.0s to 9.8s when the number of nodes increases from 1 to 7. The results provide a reference for credit assessment in the era of big data, and give suggestions from the perspective of classification algorithm and cluster.