Search

The emergence of the Digital Twin of Earth (DTE) signifies a pivotal advancement in the field of the Digital Earth, which includes observations, simulations, and predictions regarding the state of the Earth system and its temporal evolution. While the Geospatial Data Cubes (GDCs) hold promise in probing the DTE across time, existing GDC approaches are not well-suited for real-time data ingestion and processing. This paper proposes the design and implementation of a real-time ingestion and processing approach in Geospatial Data Cube (RTGDC), achieved by ingesting real-time observation streams into data cubes. The methodology employs a publish/subscribe model for efficient observation ingestion. To optimize observation processing within the data cube, a distributed streaming computing framework is utilized in the implementation of the RTGDC. This approach significantly enhances the real-time capabilities of GDCs, providing an efficient solution that brings the realization of the DTE concept within closer reach. In RTGDC, two cases involving local and global scales were implemented, and their real-time performance was evaluated based on latency, throughput, and parallel efficiency.

Real-time online trajectory map-matching has emerged as a critical component in the era of location-based services (LBS) and intelligent transportation systems (ITS). It refers to the process of aligning a user's GPS trajectory data with the corresponding road network in real-time. This technology has significant implications for various industries and applications. As our reliance on LBS and ITS continues to grow, the demand for faster, more accurate, and more reliable trajectory map-matching methods becomes increasingly important. Contemporary online map-matching predominantly employs stream processing techniques. Based on stream processing frameworks, we propose a heterogeneous hybrid architecture for map-matching. The architecture integrates Spark Streaming and graphics processing unit (GPU) heterogeneous computing for the first time. The hidden Markov model is employed as the map-matching algorithm, and Spark Streaming serves as the distributed processing platform. We conduct map-matching experiments using a GPS taxi trajectory dataset in Beijing's Haidian District. The results demonstrate that in comparison to other analogous research, our framework's performance has increased by over ten times, possessing a superior data processing capability and lower latency. This research provides a novel approach of stream-based heterogeneous computation for processing large-scale geographic data. [ABSTRACT FROM AUTHOR]

Big Data technologies emerging day by day and are making drastic changes in various real-world applications. Traditional data mining tools adequate to process volumes of data but from past decades the rapid growth in data becomes difficult for processing. Due to continuous flow of data, data streams require additional computational processing than the traditional one. Big data stream processing considers different features of the data streams heterogeneity, scalability, fault tolerance and query optimization. Efficient implementation of these features in real-world applications using big data analytics is a challenging job during data storage, processing, and analysis phases. Therefore, the proposed model FRTSPS is a generic architecture which is influenced by popular big data processing Lambda architecture, based on distributed computing platform. The architecture using open-source platform Apache Flink for doing data processing. Flink is a popular platform for processing historical and stream data flows at once parallelly. Its stateful streaming can obtain more scalability and flexibility along with high throughput and low latency than the remaining stream processing programming models. [ABSTRACT FROM AUTHOR]

Focused on the problem that the stream computing platform was suffering from the high energy consumption and low efficiency due to the lack of consideration for energy efficiency in designing process, an energy-efficient optimization strategy based on elastic data migration in big data streaming platform (EEDM-BDSP) was proposed.Firstly, models of the load prediction and the resource judgment were set up, and the load prediction algorithm was designed, which predicted the load tendency and determine node resource occupancy, so as to find nodes of resource overload and redundancy.Secondly, models of the resource constraint and the optimal data migration were set up, and the optimal data migration algorithm was proposed, which data migration for the purpose of improving node resource utilization.Finally, model of the energy consumption was set up to calculate the energy consumption saved by the cluster after data migration.The experimental results show that the EEDM-BDSP changes node resources in the cluster can responded on time, the resource utilization and the energy-efficient are improved.

Focused on the problem that the stream computing platform was suffering from the high energy consumption and low efficiency due to the lack of consideration for energy efficiency in designing process, an energy-efficient optimization strategy based on elastic data migration in big data streaming platform (EEDM-BDSP) was proposed. Firstly, models of the load prediction and the resource judgment were set up, and the load prediction algorithm was designed, which predicted the load tendency and determine node resource occupancy, so as to find nodes of resource overload and redundancy. Secondly, models of the resource constraint and the optimal data migration were set up, and the optimal data migration algorithm was proposed, which data migration for the purpose of improving node resource utilization. Finally, model of the energy consumption was set up to calculate the energy consumption saved by the cluster after data migration. The experimental results show that the EEDM-BDSP changes node resources in the cluster can responded on time, the resource utilization and the energy-efficient are improved. [ABSTRACT FROM AUTHOR]

Focused on the problem of unbalanced computing load distribution and low resource utilization in the native scheduling mechanism of big data streaming computing platform, a load balancing strategy based on Tabu search algorithm in heterogeneous environments is proposed and applied to the Apache Flink platform. Firstly, this strategy sets up a job topology model and abstracts the topology of streaming computing jobs as a directed acyclic graph.Therefore, each task slot becomes a node, which lays the foundation for performance evaluation of computing nodes.Secondly, the method imports the performance evaluation model to nodes with performance weights in the directed acyclic graph, and obtains the performance of the nodes through normalization processing; then the evaluation parameters are passed into the Tabu Search for job path optimization, so as to obtain the optimal job path. Finally, by using the CustomizationWrapper interface, this strategy allocates data to the nodes included in the optimal job path and completes the balancing of computational load. The algorithm then passes evaluation parameters into the tabu scheduling algorithm for job path optimization, thereby obtaining the optimal job path. The experimental results show that the load balancing strategy optimized by the Tabu scheduling algorithm reduces the average computing latency by 10-20ms compared to the native Flink platform. The strategy significantly improves resource utilization, and increases average throughput by about 15%. This effectively proves the effectiveness and optimization effect of the load balancing strategy. [ABSTRACT FROM AUTHOR]

Load imbalance severely affects cluster performance, and the polarization of resources due to load skewing leads to further worsening of system throughput and latency problems. The proliferation of tasks to be processed in the big data era leads to more severe load skewing. How to cope with the surge of skewed data stream in the context of big data is a new challenge now. In this paper, we propose a coordinated load balancing strategy on skewed data streams (referred to as St-Stream), which is a two-tier hierarchical system for handling data streams. The proposed strategy is characterized by performing a migration pairing strategy for resources at the task allocation stage by cutting and moving out the tasks of high-load nodes in a hierarchical manner, and the moved-out operators are placed in the routing table, and the routing table operators are moved out to these nodes sequentially according to the tasks required by low-load nodes. We further design a two-tier coordination scheme for the resource allocation problem, which can adjust the skewed load from within the nodes and then dynamically restore the balance between the nodes. We implemented St-Stream on Apache Storm, which achieves a 21% coordination in processing CPU utilization, a 17.6% reduction in latency, and a 0.3 improvement in load balance recovery compared to the baseline design. Our experimental results demonstrate that the proposed load balancing strategy better balances the cluster load and improves the performance of the stream processing system. [ABSTRACT FROM AUTHOR]

Urban geospatial monitoring is a dynamic early warning in the process of urban development. It reflects urban spatial changes from a geospatial perspective. Modern urban digital governance must use spatial data infrastructures, integrate multisource data, and implement dynamic realtime monitoring in 3D space. To meet the real-time monitoring requirements of urban geographic space in 3D environments, the rapid construction of 3D scenes, the rapid processing of monitoring information, and dynamic process simulation have become key challenges. This paper presents an approach to urban geospatial monitoring that combines a sensor web and a high-performance computing infrastructure. The approach leverages Apache Spark and stream computing to transform the traditional processing algorithm into a data retrieval and analysis process supported by high-performance computing, which drives the rapid construction of 3D scenes and the dynamic calculation of sensor web observations. The effectiveness of this method is demonstrated through a case of urban waterlogging monitoring. [ABSTRACT FROM AUTHOR]

In a big data stream computing environment, the arrival rate of data streams usually fluctuates over time, posing a great challenge to the elasticity of system. The performance of stream computing system is crucial, especially when dealing with unbounded and fluctuating data streams. Most prior studies have primarily focused on one or two aspects to enable elasticity, often lacking prompt and comprehensive performance optimization. This limitation could lead to a tuning bottleneck, preventing the system's performance from consistently reaching its optimal state. Additionally, many stream computing systems are not intelligently adaptive in real time due to the challenges of manual parameter reconfiguration for fluctuating streams. To better address these issues, we propose a framework named Sgp-Stream, which orchestrates scheduling, grouping and parallelism (Sgp). To enhance the system performance. We conduct the following research: (1) Running experiments to evaluate the impact of different factors such as scheduling, grouping and parallelism on system performance. Results show that factors at a single level usually have an upper limit on tuning system performance, and better overall performance can be achieved by coordinating multi-level factors. (2) Establishing quantitative models for stream application that consider computational cost and communication cost, multi-dimensional featured data stream, data center resources, and latency & throughput performance. (3) Demonstrating the effectiveness of the proposed runtime-aware data stream grouping based on smooth weighted polling, elastic adaptive scheduling based on Linear Deterministic Greedy and elastic scaling strategy based on Gradient Descent in Sgp-Stream, for continuous performance optimization.(4) Evaluating the application latency, throughput and resource utilization objectives using a real-world elastic stream computing system and twitter data set. Experimental results show that, compared to existing state-of-the-art works, the proposed Sgp-Stream outperforms them by reducing latency by 26%–48%, improving throughput by 14%–20%, and increasing resource utilization rate by 15%–21%, especially under increasing data stream input rates. • Experiments show single-level factor are insufficient for optimal system performance. • Establishment the DAG, data stream and resource model from quantitative perspective. • Grouping, scheduling and elastic scaling are coordinated optimization strategies. • Implementation of the prototype Sgp-Stream and its performance evaluation. [ABSTRACT FROM AUTHOR]

Data integrity is a prerequisite for ensuring data availability of IoT data and has received extensive attention in the field of IoT big data security. Stream computing systems are widely used in the field of IoT for real-time data acquisition and computing. However, the real-time, volatility, suddenness, and disorder of stream data make data integrity verification difficult. According to the survey, there is no mature and universal solution. To solve this issue, we constructed a data integrity verification algorithm scheme of the stream computing system (S-DIV) by utilizing homomorphic message authentication code and pseudo-random function security assumption. Furthermore, based on S-DIV, an external data integrity tracking and verification system is constructed to track and analyze the message data stream in real time. By verifying the data integrity of message during the whole life cycle, the problem of data corruption or data loss can be found in time, and error alarm and message recovery can be actively implemented. Then, we conduct the formal security analysis under the standard model and, finally, implement the S-DIV scheme in simulation environment. Experimental results show that the scheme can guarantee data integrity in an acceptable time without affecting the efficiency of the original system. [ABSTRACT FROM AUTHOR]

The work presented in this paper proposes a novel approach to tracking a specific vehicle over the video streams published by the collaborating traffic surveillance cameras. In recent years, smart, effective transportation systems and intelligent traffic management applications are among the topics that have been given importance by various institutions. Developing a scalable, fault-tolerant, and resilient traffic monitoring system that retrieves video chunks with the desired query is challenging. For these challenging problems, stream processing and data retrieval systems have been developed over the years. However, there are still existing shortcomings between users and retrieval systems. This paper investigates the problem of retrieving video chunks by key-value query based on publish/subscribe model. Thus, we propose a hybrid of an asynchronous and synchronous communication mechanism for the Event-Based Microservice framework. We aim to develop generic techniques for better utilization of existing platforms. In the proposed framework, (i) first of all, microservices detect vehicles and extract their type, color, and speed features, and stored them in the metadata repository. (ii) Microservices publish each feature as events (iii) Other microservices self-join subscribe to those events, which leads to more events being published by combing all the possibilities: type-color, type-speed, color-speed, and type-color-speed. Finally, (iv) the system visualizes the query result and system status in real-time. When the user has selected color or/and a type or/and a speed feature, the system will return the best-matched vehicles without re-processing the videos. Experimental results show that our proposed system filters messages in real-time and supports easy integration of new microservices with the existing system.

As one of the most popular high-performance data processing technology, existing task and resource scheduling strategies for stream computing platforms are suffering from the problem of triggering hysteresis, which seriously affects the cluster performance. To address this problem, the idea of performance prediction based on timeline series data is proposed. Firstly, the performance variation rule of the stream processing platform is analyzed, which provides a basis for proposing the performance prediction model. Secondly, the basic topology paradigm, periodic load prediction model, and real-time throughput prediction model are proposed as the theoretical foundation for performance prediction. Thirdly, the performance prediction algorithm is proposed to predict the variation trend of the processing load and throughput of the cluster. The processing load is predicted periodically while the throughput of the cluster is predicted in a real-time manner. Finally, the performance evaluation algorithm is proposed to evaluate the cluster performance based on the prediction results and trigger the corresponding scheduling strategies in advance. The experimental results showed that the prediction accuracy of the proposed algorithm meets the requirements in practical applications. Meanwhile, the proposed method improves the performance of stream computing performance by triggering the scheduling strategies in advance.

In order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources,the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) was proposed.Firstly,the load prediction model was set up as the foundation to propose the load prediction algorithm and predict the variation tendency of the processing load.Secondly,the resource judgment model was set up to identify the performance bottleneck and resource redundancy of the cluster while the resource scheduling algorithm was proposed to draw up the resource rescheduling plan.Finally,the online load migration algorithm was proposed to execute the resource rescheduling plan and migrate processing load among nodes efficiently.The experimental results show that the strategy provides better performance promotion in the application with drastically fluctuating processing load.The scale and resource configuration of the cluster responded to the variation of processing load in time and the communication overhead of the load migration was reduced effectively.

Path planning is a key problem to be solved for mobile robot to realize autonomous navigation. It is a typical computing intensive task, and high computing capacity is needed. The computing power carried by the mobile robot is difficult to support the calculation of path planning, and the traditional cloud computing model cannot meet the real-time requirement of path planning. In order to solve the problem of insufficient computing power and improve the execution efficiency and real-time performance, a real-time computing framework based on edge-cloud collaborative computing is constructed for path planning. In each control decision cycle, the mobile robot as the edge acquires the sensing data and transmits it to the cloud. By stream computing, the cloud plans the path in real-time. The edge integrates the planned path from the cloud and the partial obstacle avoidance result from the edge as a path sequence. The final path sequence is sent to the motion control layout, and drives the mobile robot to the target. By the edge-cloud collaborative computing, the computing capability of the edge is extended. By taking use of high real-time performance of stream computing, the proposed algorithm improves the efficiency of path planning. By taking use of the storage capacity of the cloud, environmental memory is realized and the problem of local traps is solved. Simulation experiment results in different environments show that the planned path by the proposed algorithm gets a higher path quality and shorter execution time comparing the other several traditional path planning algorithms. Experiments in real environment verify the feasibility and effectiveness of the algorithm. [ABSTRACT FROM AUTHOR]

The physiological signal acquisition and analyzing are important for intelligent health services, human–computer interaction and other applications. Due to the computing power limitation of terminal devices, many analyzing methods of physiological signals are in offline mode. However, in many applications, physiological signal should be analyzed in real time. To overcome this problem, a real-time physiological signal acquisition and analysis method based on fractional calculus and stream computing is proposed. Mobile terminals read the physiological data from sensors and upload them to the stream computing platform. A fractal index is used to estimate the physiological status. Based on the stream computing platform, this index is calculated by distributed parallel computing. The experiment results show this method can distinguish the heart health status and reflect driver mental status to some extent. [ABSTRACT FROM AUTHOR]

In recent years,with the rapid development of FinTech,digital finance has flourished and brought huge positive effect on society.Meanwhile,new risks have been introduced into banks.For example,the black production related to network security has experienced explosive growth,and telecommunication network fraud has caused property losses to the public.In the era of digital finance,the commercial banks have not only ushered in new opportunities and dynamics,but also faced new challenges and requirements for digital transformation.As a result,e-finance has become a new battlefield.With this context,a real-time anti-fraud machine learning model based on high-dimensional transaction behavior portrait through enhanced RFM feature-derivation and machine learning modeling was established in this paper.Relying on the new technologies such as big data,stream computing,a model application solution to real-time risk control was formed including systematic deployment,model application strategies and iterative model optimization.Through practical observation,the AUC of the model reaches 0.972,which provides a keen insight into fraud risk,realizes millisecond-level risk identification,and promotes risk control ability of e-finance significantly.

In order to solve the problem that the load of big data stream computing platform is increasing with fluctuation while the cluster was not able to rescale efficiently,the Flow-network based auto rescale strategy for Flink was proposed.Firstly,the flow-network model was set up and the capacity of each edge that was calculated by self-learning algorithm.Secondly,the bottleneck of the cluster was acquired by maximum-flow algorithm and the resource rescheduling plan was drawn up.Finally,the resource rescheduling plan was executed and the stateful data was migrated efficiently by the data migration algorithm based on the strategy of data partitioning by bulk and bucket.The experimental results show that the strategy can effectively provide performance promotion in the application with complex stateful data.It improved the throughput of the cluster and reduced the time overhead of the data migration on the premise of satisfying the latency constrain of the application,which means that the strategy promotes the scalability of the cluster efficiently.

Abstract Recently, big data streams have become ubiquitous due to the fact that a number of applications generate a huge amount of data at a great velocity. This made it difficult for existing data mining tools, technologies, methods, and techniques to be applied directly on big data streams due to the inherent dynamic characteristics of big data. In this paper, a systematic review of big data streams analysis which employed a rigorous and methodical approach to look at the trends of big data stream tools and technologies as well as methods and techniques employed in analysing big data streams. It provides a global view of big data stream tools and technologies and its comparisons. Three major databases, Scopus, ScienceDirect and EBSCO, which indexes journals and conferences that are promoted by entities such as IEEE, ACM, SpringerLink, and Elsevier were explored as data sources. Out of the initial 2295 papers that resulted from the first search string, 47 papers were found to be relevant to our research questions after implementing the inclusion and exclusion criteria. The study found that scalability, privacy and load balancing issues as well as empirical analysis of big data streams and technologies are still open for further research efforts. We also found that although, significant research efforts have been directed to real-time analysis of big data stream not much attention has been given to the preprocessing stage of big data streams. Only a few big data streaming tools and technologies can do all of the batch, streaming, and iterative jobs; there seems to be no big data tool and technology that offers all the key features required for now and standard benchmark dataset for big data streaming analytics has not been widely adopted. In conclusion, it was recommended that research efforts should be geared towards developing scalable frameworks and algorithms that will accommodate data stream computing mode, effective resource allocation strategy and parallelization issues to cope with the ever-growing size and complexity of data.

This paper focuses on the time efficiency for machine vision and intelligent photogrammetry, especially high accuracy on-board real-time cloud detection method. With the development of technology, the data acquisition ability is growing continuously and the volume of raw data is increasing explosively. Meanwhile, because of the higher requirement of data accuracy, the computation load is also becoming heavier. This situation makes time efficiency extremely important. Moreover, the cloud cover rate of optical satellite imagery is up to approximately 50%, which is seriously restricting the applications of on-board intelligent photogrammetry services. To meet the on-board cloud detection requirements and offer valid input data to subsequent processing, this paper presents a stream-computing of high accuracy on-board real-time cloud detection solution which follows the “bottom-up” understanding strategy of machine vision and uses multiple embedded GPU with significant potential to be applied on-board. Without external memory, the data parallel pipeline system based on multiple processing modules of this solution could afford the “stream-in, processing, stream-out” real-time stream computing. In experiments, images of GF-2 satellite are used to validate the accuracy and performance of this approach, and the experimental results show that this solution could not only bring up cloud detection accuracy, but also match the on-board real-time processing requirements.

We introduce Tiered Sampling, a novel technique for estimating the count of sparse motifs inmassive graphs whose edges are observed in a stream. Our technique requires only a single pass on the data and uses a memory of fixed size M, which can be magnitudes smaller than the number of edges. Our methods address the challenging task of counting sparse motifs--sub-graph patterns--that have a low probability of appearing in a sample of M edges in the graph, which is the maximum amount of data available to the algorithms in each step. To obtain an unbiased and low variance estimate of the count, we partition the available memory into tiers (layers) of reservoir samples. While the base layer is a standard reservoir sample of edges, other layers are reservoir samples of sub-structures of the desired motif. By storing more frequent sub-structures of the motif, we increase the probability of detecting an occurrence of the sparse motif we are counting, thus decreasing the variance and error of the estimate. While we focus on the designing and analysis of algorithms for counting 4-cliques, we present a method which allows generalizing Tiered Sampling to obtain high-quality estimates for the number of occurrence of any sub-graph of interest, while reducing the analysis effort due to specific properties of the pattern of interest. We present a complete analytical analysis and extensive experimental evaluation of our proposed method using both synthetic and real-world data. Our results demonstrate the advantage of our method in obtaining high-quality approximations for the number of 4 and 5-cliques for large graphs using a very limited amount of memory, significantly outperforming the single edge sample approach for counting sparse motifs in large scale graphs. [ABSTRACT FROM AUTHOR]

Aiming at the problem of lacking efficient distributed fault tolerant mechanism for data explosion in the distributed distribution power automation system, based on the record-level fault tolerance of a resilient distributed dataset (RDD) and micro-batch computing, a new lineage chain mark (LCM) fault-tolerant method is proposed. Using the distribution power network monitoring dataset as cache object, the programming model based on the micro-batch processing framework is comprehensively used to design the topology example of the distribution power network monitoring data, which can shorten the derivative chain of RDD. This paper takes the monitoring data in a dispatching and monitoring system of the Shijiazhuang-Dezhou distribution power network as the data source, which has deployed a flow calculation test cluster of one primary node and three working nodes. Taking the most frequent fault of single data node failure condition as an example, the test results based on 3 million monitoring data in RDD show that, the CPU usage volatility of this model in cluster nodes was less than 1.5%, the network consumption of read and write execution is decreased by about 3.1% and 5.0%, respectively, compared with the original model, and the disk usage was reduced by 4.2%; the iterative test shows that the number of iterations is positively correlated with the computational time, which is consistent with the calculation formula for the worst execution time of the lineage chain labeling model. When the number of iterations is more than 400, the greater the number of iterations, the more significant the LCM method improves the performance of the calculation. The results show that the long pedigree chain can be cut off by using the pedigree chain marker sequence, so that multi-computer parallel processing can be carried out in the nodes, which reduces the computational overhead of iterative operation and effectively improves the performance of fault-tolerant computing.

Focused on the problem that traditional energy-efficient strategies never consider about the real time of data processing and transmission,models of directed acyclic graph,parallelism of instance,resource allocation for task and critical path were set up based on the features of data stream processing and the structure of storm cluster.Meanwhile,the WNDVR-storm (energy-efficient strategy for work node by dram voltage regulation in storm) was proposed according to the analysis of critical path and system performance,which included two energy-efficient algorithms aiming at whether there were any work nodes executing on the non-critical path of a topology.Finally,the appropriate threshold values fit for the CPU utilization of work node and the volume of transmitted data were determined based on the data processing and transmission constraints to dynamically regulate the DRAM voltage of the system.The experimental result shows that the strategy can reduce energy consumption effectively.Moreover,the fewer constraints are,the higher energy efficiency is.

As a new platform in big data stream computing, Apache Heron ignores the difference in communication modes between task instances and the unbalance of processing load among nodes, which leads to the decline system performance. To address the problem, this paper designed the model of node resource limitation, the model of communication overhead optimization and the model of data stream relationships among instances, as the foundation to propose the TUR-Heron. The strategy was composed of the node resource limitation algorithm and the instance reallocation algorithm. By judging the criteria for instance reallocation and executing instance reallocation algorithm, this strategy transformed the inter-node data streams into intra-node data streams and minimized the communication overhead of the system. The experimental results show that under the three sets of benchmarks, TUR-Heron reduces the communication overhead between nodes and the response latency of the system compared with the default scheduling strategy, and improves the balance of resource utilization of computing nodes. [ABSTRACT FROM AUTHOR]

In this article, we introduce an automatic stream computing reoptimization flow from ASICs to field-programmable gate arrays (FPGAs). Complex VLSI designs need to be prototyped and/or emulated on FPGAs. The main problem that we address in this article is that configurations optimized when targeting ASICs are often, as we will show in this article, highly un-optimal when remapped onto an FPGA. Thus, this article proposes a method to first generate a variety of dataflow configurations targeting an ASIC given multiple behavioral descriptions for high-level synthesis (HLS) and then, based on a compositional predictive model, automatically reoptimize the dataflow when mapped onto an FPGA. The experimental results show that our proposed method works well and that it is very fast. [ABSTRACT FROM AUTHOR]

Copyright of Journal on Communication / Tongxin Xuebao is the property of Journal on Communications Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In the last years, pattern-based programming has been recognized as a good practice for efficiently exploiting parallel hardware resources. Following this approach, multiple libraries have been designed for providing such high-level abstractions to ease the parallel programming. However, those libraries do not share a common interface. To pave the way, GrPPI has been designed for providing an intermediate abstraction layer between application developers and existing parallel programming frameworks like OpenMP, Intel TBB or ISO C++ threads. On the other hand, FastFlow has been adopted as an efficient object-based programming framework that may benefit from being supported as an additional GrPPI backend. However, the object-based approach presents some major challenges to be incorporated under the GrPPI type safe functional programming style. In this paper, we present the integration of FastFlow as a new GrPPI backend to demonstrate that structured parallel programming frameworks perfectly fit the GrPPI design. Additionally, we also demonstrate that GrPPI does not incur in additional overheads for providing its abstraction layer, and we study the programmability in terms of lines of code and cyclomatic complexity. In general, the presented work acts as reciprocal validation of both FastFlow (as an efficient, native structured parallel programming framework) and GrPPI (as an efficient abstraction layer on top of existing parallel programming frameworks). [ABSTRACT FROM AUTHOR]

The signaling data of telecom operators,which have a wide range of applications and great of values,is big data in real sense.In order to serve the real-time marketing scenarios better,the real-time processing and analysis ability of signaling data must be improved effectively.Firstly,the scheme of signaling collection and processing based on FKS was introduced,which combined the advantages of components such as Flume,Kafka and Spark Streaming,and played an important supporting role for operators to build real-time decision center.Furthermore,the Kafka monitoring program for topic consumption of consumer groups was proposed,which provided effective means for warning faults of real-time flow collection.The monitoring solution could effectively reduce the message queue congestion,avoid the risk of data loss,and also could obtain good application effects in practice.

Traditional environmental model simulations often use archived data as inputs. Recent advancement of Sensor Web technologies in Spatial Data Infrastructures (SDIs) allows real-time observations to be fed into models to generate "live" models. A key challenge is how to efficiently process observation streams in models, which is particularly important in time-critical cases like disaster management. This paper presents an observation stream computing model for live modelling, which couples Sensor Web and models in stream computing environment to provide timely decision-support information. Observation Streams are proposed as information models to deal with observation stream processing. The approach shows how MapReduce and Apache Spark stream processing can be leveraged to support coupling of observation streams and models. The approach is applied in a disaster management case, where in-situ observation streams are processed to compute the waterlogging information in near real time. The results illustrate applicability and effectiveness of the approach. • A framework for coupling Sensor Web and models in a stream computing environment. • O-Stream information model facilitates the observation stream processing in SPARK. • Case study of the live waterlogging model to support the model cloud. [ABSTRACT FROM AUTHOR]