Search

Spark is one of the most widely used systems for the distributed processing of big data. Its performance bottlenecks are mainly due to the network I/O, disk I/O, and garbage collection. Previous studies quantitatively analyzed the performance impact of these bottlenecks but did not focus on iterative algorithms. In an iterative algorithm, garbage collection has more performance impact than other workloads because the algorithm repeatedly loads and deletes data in the main memory through multiple iterations. Spark provides three caching mechanisms which are “disk cache,” “memory cache,” and “no cache” to keep the unchanged data across iterations. In this paper, we provide an in-depth experimental analysis of the effect of garbage collection on the overall performance depending on the caching mechanisms of Spark with various combinations of algorithms and datasets. The experimental results show that garbage collection accounts for 16–47% of the total elapsed time of running iterative algorithms on Spark and that the memory cache is no less advantageous in terms of garbage collection than the disk cache. We expect the results of this paper to serve as a guide for the tuning of garbage collection in the running of iterative algorithms on Spark.

We report a simple method to produce work-function-tuned graphene nanosheets based on the nucleophilic substitution of the epoxy groups on graphene oxide. The electrical property of the graphene oxide is controlled dramatically, which results in the apparent work functions in a broad range between 3.73 eV and 5.1 eV, by attaching various functional groups on the graphene surface. As a proof of concept, we successfully demonstrated organic field effect transistors incorporating the functionalized graphene nanosheet interlayers. Here, when nanosheets were applied in an organic transistor as the interlayer material between electrodes and organic channel, the device performance was significantly improved. Our approach can be utilized to increase the performance and the flexibility of various advanced carbon-material-based hybrid electrical devices.

MapReduce is the most popular framework for distributed processing. Recently, the scalability of data mining and machine learning algorithms has significantly improved with help from MapReduce. However, MapReduce does not handle iterative algorithms very efficiently. The problem is that many data mining and machine learning algorithms are iterative by nature. In order to overcome the limitations of MapReduce, many advanced distributed systems have been developed, including HaLoop, iMapReduce, Twister, and Spark. In this paper, we identify and categorize the limitations of MapReduce in handling iterative algorithms, and then, experimentally investigate the consequences of these limitations by using the most flexible and stable distributed system, Spark. According to our experiment results, the network I/O overhead was the primary factor that affected system performance the most. The disk I/O overhead also affected system performance, but it was less significant than the network I/O overhead. For the synchronization overhead, it affected system performance only when the static data was not cached.

Community detection from social network data gains much attention from academia and industry since it has many real-world applications. The Girvan-Newman (GN) algorithm is a divisive hierarchical clustering algorithm for community detection, which is regarded as one of the most popular algorithms. It exploits the concept of edge betweenness to divide a network into multiple communities. Though it is being widely used, it has limitations in supporting large-scale networks since it needs to calculate the shortest path between every pair of vertices in a network. In this paper, we develop two parallel versions of the GN algorithm to support large-scale networks. First, we propose a new algorithm, which we call Shortest Path Betweenness MapReduce Algorithm (SPB-MRA), that utilizes the MapReduce model. Second, we propose another new algorithm, which we call Shortest Path Betweenness Vertex-Centric Algorithm (SPB-VCA), that utilizes the vertex-centric model. An approximation technique is also developed to further speed up community detection processes. We implemented SPB-MRA using Hadoop and SPB-VCA using GraphChi, and then evaluated the performance of SPB-MRA on Amazon EC2 instances and that of SPB-VCA on a single commodity PC. The evaluation results showed that the elapsed time of SPB-MRA decreased almost linearly as the number of reducers increased, SPB-VCA outperformed SPB-MRA just on a single PC by 4-6 times, and the approximation technique introduced negligible errors.

This work proposes sliding window fall detection match (SW-FDM), a rule-based fall detection method based on event pattern matching from human body posture event streams. Fall and post-fall (long lie) rules are expressed as patterns, and complex event processing (CEP) systems are adopted to quickly find these patterns. They can be detected with event selection strategies such as Skip Till Next Match and Skip Till Any Match. However, existing strategies generate either duplicate or missing alarms; even worse, their processing cost is very high when the size of event streams is large. Since SW-FDM uses a concept of sliding window, it is able to detect correct matches constantly and reduce the processing cost without duplicate computation. The experiments demonstrate that SW-FDM results in both higher accuracy and efficiency. Also, it is shown that the improvement of efficiency becomes greater as the data size increases, which is an indeed preferable property.

Since the development of MapReduce, there have been several efforts to extend data mining and machine learning algorithms for MapReduce. Many of those algorithms are iterative by nature. In order to process them efficiently, Spark as well as research prototypes such as HaLoop, iMapReduce, and Twister are proposed with solutions to iterative computation. In this paper, we thoroughly examine the pros and cons of each system.