Search

This paper presents an in-depth study and analysis of large datasets of mixed and attribute features under Spark using a large dataset clustering algorithm. The classical algorithm K-means based on division and the density-based clustering algorithm DPC, which has become more popular in recent years, are selected as the research objects of this paper. Secondly, the original K-means algorithm is improved by combining holdout validation and K-means++ method to address the shortcomings of the K-means algorithm that the number of class clusters K needs to be set in advance and the initial class cluster centers are chosen randomly, which leads to unstable iterations and slow convergence of clustering results. The similarity matrix will be continuously updated during the iterative process. It mainly refers to the process of dividing objects into multiple classes according to the degree of similarity between objects. After the division, the objects within the class are like each other, while the objects between the classes are different from each other. The comparison experiments of the improved algorithm before and after the MovieLens dataset are conducted to verify that the new algorithm has better performance in terms of clustering accuracy and efficiency. Again, to address the drawback that the clustering results in the DPC algorithm rely heavily on the subjective selection of the truncation distance parameter cd, and it is difficult to handle datasets with complex distribution and large density variation, the algorithm can generate the optimal cd adaptively by combining K-nearest neighbors and introducing the distance comparison quantity, which has a better performance by considering the overall and local distribution of the data. The feasibility of the improved method is verified by validating the algorithm with artificial datasets and UCI datasets as well as separation tests. Finally, the parallelized design and implementation of the improved K-means algorithm and CDPC-KNN algorithm are completed by building a Spark clustering environment, and the parallelized algorithm is verified to have much better data processing capability and be more adaptable to the clustering analysis of large-scale data by comparing algorithm string parallelism experiments. [ABSTRACT FROM AUTHOR]

Cloud computing provides multiple services such as computational services, data processing, and resource sharing through multiple nodes. These nodes collaborate for all prementioned services in the data center through the head/leader node. This head node is responsible for reliability, higher performance, latency, and deadlock handling and enables the user to access cost-effective computational services. However, the optimal head nodes' selection is a challenging problem due to consideration of resources such as memory, CPU-MIPS, and bandwidth. The existing methods are monolithic, as they select the head nodes without taking the resources of the nodes. Still, there is a need for the candidate node which can be selected as a head node in case of head node failure. Therefore, in this paper, we proposed a technique, i.e., Head Node Selection Algorithm (HNSA), for optimal head node selection from the data center, which is based on the genetic algorithm (GA). In our proposed method, there are three modules, i.e., initial population generation, head node selection, and candidate node selection. In the first module, we generate the initial population by randomly mapping the task on different servers using a scheduling algorithm. After that, we compute the overall cost and the cost of each node based on resources. In the second module, the best optimal nodes are selected as a head node by applying the genetic operations such as crossover, mutation, and fitness function by considering the available resources. In the selected optimal nodes, one node is chosen as a head node and the other is considered as a candidate node. In the third module, the candidate node becomes the head node in the case of head node failure. The proposed method HNSA is compared against the state-of-the-art algorithms such as Bees Life Algorithm (BLA) and Heterogeneous Earliest Finished Time (HEFT). The simulation analysis shows that the proposed HNSA technique performs better in terms of execution time, memory utilization, service level sgreement (SLA) violation, and energy consumption. [ABSTRACT FROM AUTHOR]

In view of the miniaturization and decentralization characteristics of agricultural equipment factories in China, agricultural equipment manufacturing is well suited to the cloud manufacturing model, but there is no specific research on cloud services optimization for it. To fill the research gap, a cloud service optimization method is proposed in this paper. For the optimization model, the dynamic coefficient strategy and the reliability feedback update strategy are added to the mathematical model to strengthen the applicability of farming season. As optimization algorithm, a dynamic artificial ant-bee colony algorithm (DAABA) based on artificial ant colony algorithm and bee colony algorithm is presented. The optimal fusion evaluation strategy is used to save optimization time by reducing the useless iteration, and the iterative adjustment threshold strategy is adopted to improve the accuracy of cloud service by increasing the size of bee colony. Finally, the performance of DAABA is verified to be more superior by comparing with other algorithms. [ABSTRACT FROM AUTHOR]