Search

The early Internet of Things stream processing platforms were mainly designed to collect and display real‐time raw sensor measurements. Data often has to go through several phases of processing to lead to actionable automatic or human decision making. This chapter discusses five main operations performed on streaming data: compression, dimensionality reduction, summarization, learning, and visualization. It distinguishes two main types of stream data processing systems. The first type, also called a data stream management system, is based on relational database principles and introduced the concept of continuous queries. The second type does not enforce a relational view and enables the creation of custom operators. Based on batch and stream processing paradigms, two data processing architectures emerged: the Lambda architecture, which enables both batch and stream processing, and the simpler Kappa architecture, which enables stream processing.

Large-scale industrial internet of things (IIoT) applications usually access distributed equipment where high-volume sensor streams are processed. The building of scalable analytic queries and models over such streams could potentially enhance various industrial processes management tasks, e.g., distribution, delivery, and predictive online maintenance. To enable real-time and historical analytics over distributed IIoT applications, we have combined an edge data stream management system (EDSMS), sa.engine, with the highly distributed NoSQL database MongoDB. For supporting advanced analytics and high-volume stream injection into MongoDB, we integrated an extended query processing (EQP) system with sa.engine and MongoDB. This work demonstrates how EQP provides a holistic data management solution for IIoT based on a use case for sound anomaly detection of distributed equipment.

The increasing availability of indoor and outdoor location sensors raises the interest for understanding the mobility in different places, e.g., in museums, train stations, or cities. One of the highly used approaches for understanding the mobility of individuals is to detect staypoints from trajectories. Although offline detection of staypoints fits best for long-term planning, some applications require online staypoint detection from trajectories, like online recommendation systems. Latency plays an important role here. In this paper, we propose a stream-based approach for staypoint detection, which can be realized by applying a Data Stream Management System. We claim that the proposed approach can detect staypoints with low latency from the high-frequency location update streams. To evaluate our approach, we compare it with a batch-based approach on real data from an indoor tracking system and Geolife dataset. The results demonstrate that the online approach detects staypoints with much lower latency compared to traditional approaches like offline and batch processing. Moreover, we prove that the accuracy of the stream-based approach is similar to the batch-based approach.

the number of connected objects used in our daily lives (smartphones, smart watches, etc.) is constantly increasing. These smart and connected objects usually have a unique identifier and communication, processing and networking capabilities. These smart devices can connect to the Internet, creating a vast network of objects called the "Internet of Things" (IoT). Thus, they can interact (detect and act) with their physical environment.However, the IoT raises many challenges related to its very large scale and high dynamicity, as well as the large variety of the generated data. These issues require the design and development of new systems and techniques to (i) collect and store data from the different smart objects, (ii) process and analyse these data, (iii) assist managers in the decision making process through simplified information. This paper proposes a data stream management system that collect, store, manage, analyse and visualize cultural heritage data. We propose a new data management architecture based on Big Data and cloud computing technologies. The proposed architecture consists of a distributed data management system for the tourism IoT. The proposed architecture addresses the problems of deploying continuous processing tasks automatically and simplify the access to the collected data.

Quality of Service (QoS) has been identified as an important attribute of system performance of Data Stream Management Systems (DSMS). A DSMS should have the ability to allocate physical computing resources between different submitted queries and fulfil QoS specifications in a fair and square manner. System scheduling strategies need to be adjusted dynamically to utilise available physical resources to guarantee the end-to-end quality of service levels. In this paper, we present a proactive method that utilises a multi-level component profiling approach to build prediction models that anticipate several QoS violations and performance degradations. The models are constructed using several incremental machine learning algorithms that are enhanced with ensemble learning and abnormal detection techniques. The approach performs accurate predictions in near real-time with accuracy up to 85% and with abnormal detection techniques, the accuracy reaches 100%. This is a major component within a proposed QoS-Aware Self-Adapting Data Stream Management Framework.

The last decade witnessed plenty of Big Data processing and applications including the utilisation of machine learning algorithms and techniques. Such data need to be analysed under specific Quality of Service (QoS) constraints for certain critical applications. Many frameworks have been proposed for QoS management and resource allocation for the various Distributed Stream Management Systems (DSMS), but lack the capability of dynamic adaptation to fluctuations in input data rates. This paper presents a novel QoS-Aware, Self-Adaptive, Resource Utilisation framework which utilises instantaneous reactions with proactive actions. This research focuses on the load monitoring and analysis parts of the framework. By applying real-time analytics on performance and QoS metrics, the predictive models can assist in adjusting resource allocation strategies. The experiments were conducted to collect the various metrics and analyse them to reduce their dimensions and identify the most influential ones regarding the QoS and resource allocation schemes.

These Event processing systems are much used in wide variety of applications in the processing of large stream of events. The most distinguished of applications is the time-series data management system with timely processing to identify trends, pattern matches and forecast future values.The complexity of event information, coupled with the fact that historical event data is being kept in the database, requires the use of an event processing model that provides the user with high-level abstractions. In this paper, I survey the StreamEPS to help developers and researchers alike to understand the conceptual view and processing of the event processing software system. StreamEPS forms part of Complex EventProcessing (CEP), Data Stream Management System (DSMS) and Information Flow Processing (IFP) domain.

In recent years a new class of data intensive application has emerged. In these applications data is modeled best not as persistent databases, but rather as transient data streams. In this chapter we assess tidal and wave energy systems through data stream management systems. This chapter includes blocking parameters of tidal and wave energy system and also assesses such types of ocean energy systems by different types of sampling methods. At the end the financial assessment of ocean energy systems is undertaken.

State-of-the-art security frameworks have been extensively addressing security issues for web resources, agents and services in the Semantic Web. The provision of Stream Reasoning as a new area spanning Semantic Web and Data Stream Management Systems has eventually opened up new challenges. Namely, their decentralized nature, the metadata descriptions, the number of users, agents, and services, make securing Stream Reasoning systems difficult to handle. Thus, there is an inherent need of developing new security models which will handle security and automate security mechanisms to a more autonomous system that supports complex and dynamic relationships between data, clients and service providers. We plan to validate our approach on a typical application of stream data, on Wireless Sensor Networks (WSNs). In particular, WSNs for water quality monitoring will serve as a case study. The paper describes the initial findings and research plan for building a consistent security model for stream reasoning systems.

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of Things. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. The current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes specific assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. Therefore, in this survey, we study the literature and develop a classification of current methods for both parallelization and elasticity in SP systems., 37 Pages, to be published in ACM Computing Surveys

Owing to the technological advancements in Semantic Web and sensor networks, a large amount of data has been produced in association with the open data policy. However, data stream management systems that process stream data have focused on the processing of a large amount of data with little priority on data identification, integration, and external linkage. Furthermore, entity resolution is focused mainly on static database-based technologies. In this study, a real-time stream data processing architecture that can perform the integration and entity resolution of streaming-type heterogeneous input data and interlink with external data is designed. To achieve this goal, a light adapter to integrate heterogeneous data into standard scheme and blocking technique to reduce comparison candidates are applied. The implemented data adapters shows 4 times higher throughput than open source data parsers and the entity resolution results with streaming data shows similar performance with the static data sets. The proposed streaming data entity resolution architecture is expected to form the basis of data integration research that can integrate various information sources of data efficiently, enrich internal data.

Current data management systems are mainly divided into two categories: Database Management System (DBMS) and Data Stream Management System (DSMS). The increasing use of streaming analysis in modern service-based cloud applications has created an arms race among DBMS vendors to offer ever more sophisticated in-database streaming support, which requires handling the volume, variety, velocity and variability of fast data collections. Unfortunately, current solutions either only provide limited streaming analysis capacity and horizontal scalability (classic RDBMS) or trade off transaction processing for other properties (NoSQL DBMS), leading to the curse of no "one size fits all" for DBMS. In this paper, we argue that transaction processing is a relevant concept for DSMS. As a first step toward "One Size Fits All" Data Management System, we present StreamDB, which integrates transaction processing in DSMS as opposed to extending DBMS to support streams. First, we describe how StreamDB processes transactions in a streaming environment, then we compare our approach with traditional in-memory DBMS on typical transactional benchmarks. Our results show that StreamDB is advantageous in terms of throughput, scalability, and latency. Finally, we argue that the ideas present here provide insight on the development of next-generation data management systems and motivate further study of the challenges inherent in unifying DBMS and DSMS.

The development of applications for advanced Intelligent Transportation Systems (ITS), is a growing research area, pushed especially by the need to mitigate problems related to transportation inefficiency, overuse and pollution, mainly caused by the increasing of traffic congestions in big cities. Nowadays, traffic related data coming from the road infrastructure, user's cars or mobile phones, is generated in unprecedent volume and speed. Therefore, ITS face new challenges, with respect to growing demand to process large volumes of traffic data in real-time, and extract value that can be immediately used for decision making. The work presented here, proposes a scalable architecture supported by Big Data technologies, capable of processing real-time traffic data captured from 349 inductive loop counters, placed in Slovenian road network. The main goal is to propose an approach which can be adopted by national road operators, capable of monitoring, in real-time, the current status of the road network. Traffic events need to be detected in real-time and require the collection of data from several fixed sensors. To deal with data volume and velocity challenges, a data stream management system was used, able to collect traffic data that arrives and process it on a continuous (24×7) basis. Results achieved, suggest that there's a great advantage in the adoption of a data stream management system, with respect to traditional database management systems, illustrated by practical examples, providing quantitative metrics about data processing performance.

Data stream management systems (DSMSs) offer the most effective solution for processing data streams by efficiently executing continuous queries (CQs) over the incoming data. CQs inherently have different levels of criticality and hence different levels of expected quality of service (QoS) and quality of data (QoD). Adhering to such expected QoS/QoD metrics is even more important in cases of multi-tenant data stream management services. In this work, we propose DILoS, a framework that, through priority-based scheduling and load shedding, supports differentiated QoS and QoD for multiple classes of CQs. Unlike existing works that consider scheduling and load shedding separately, DILoS is a novel unified framework that exploits the synergy between scheduling and load shedding. We also propose ALoMa, a general, adaptive load manager that DILoS is built upon. By its design, ALoMa performs better than the state-of-the-art alternatives in three dimensions: (1) it automatically tunes the headroom factor, (2) it honors the delay target, (3) it is applicable to complex query networks with shared operators. We implemented DILoS and ALoMa in our real DSMS prototype system (AQSIOS) and evaluate their performance for a variety of real and synthetic workloads. Our experimental evaluation of ALoMa verified its clear superiority over the state-of-the-art approaches. Our experimental evaluation of the DILoS framework showed that it (a) allows the scheduler and load shedder to consistently honor CQs' priorities, (b) significantly increases system capacity utilization by exploiting batch processing, and (c) enables operator sharing among query classes of different priorities while avoiding priority inversion, i.e., a lower-priority class never blocks a higher-priority one.

This article presents a new, original application of modern information and communication technology to provide effective real-time dissemination of air quality information and related health risks to the general public. Our on-line subsystem for urban real-time air quality monitoring is a crucial component of a more comprehensive integrated information system, which has been developed by the Institute for Medical Research and Occupational Health. It relies on a StreamInsight data stream management system and service-oriented architecture to process data streamed from seven monitoring stations across Zagreb. Parameters that are monitored include gases (NO, NO2, CO, O3, H2S, SO2, benzene, NH3), particulate matter (PM10 and PM2.5), and meteorological data (wind speed and direction, temperature and pressure). Streamed data are processed in real-time using complex continuous queries. They first go through automated validation, then hourly air quality index is calculated for every station, and a report sent to the Croatian Environment Agency. If the parameter values exceed the corresponding regulation limits for three consecutive hours, the web service generates an alert for population groups at risk. Coupled with the Common Air Quality Index model, our web application brings air pollution information closer to the general population and raises awareness about environmental and health issues. Soon we intend to expand the service to a mobile application that is being developed.

Sensor and communication development has led to the development of new types of applications. Classic database data storage becomes inadequate when data streams arrive from multiple sensors. Then, data querying and result presentation are not efficient. The desired results are obtained with a delay, and the database is filled with a large amount of unnecessary data. To adequately support the above applications, Data Stream Management System (DSMS) applications are needed. DSMSs provide real-time data stream processing. In this paper, a client-server system is presented with DSMS realized on the Java WebDSMS application server side. WebDSMS functionalities are tested with simulated data and in real-life usage. [Projekat Ministarstva nauke Republike Srbije, br. TR32043]

International audience; As distributed IoT applications become larger and more complex, the pure processing of raw sensor and actuation data streams becomes impractical.Instead, data streams must be fused into tangible facts and these pieces of information must be combined with a background knowledge to infer new pieces of knowledge. And since many IoT applications require almost real-time reactivity to stimulus of the environment, such information inference process has to be performed in a continuous, on-line manner. This paper proposes a new semantic model for data stream processing and real-time reasoning based on the concepts of Semantic Stream and Fact Stream, as a natural extension of Complex Event Processing (CEP) and RDF (graph-based knowledge model). The main advantages of our approach are that: (a) it considers time as a key relation between pieces of information; (b) the processing of streams can be implemented using CEP; (c) it is general enough to be applied to any Data Stream Management System (DSMS). Last, we will present challenges and prospects on using machine learning and induction algorithms to learn abstractions and reasoning rules from a continuous data stream.

In this paper, to analyze end-to-end timing behavior in heterogeneous processor and network environments accurately, we adopt a heterogeneous selection value on communication contention (HSV_CC) algorithm, which can synchronize tasks and messages simultaneously, for stream processing distribution. In order to adapt HSV_CC to automotive data stream management system (DSMSs), one must address three issues: (i) previous task and message schedules might lead to less efficient resource usages; (ii) the conventional method to determine the task scheduling order may not be best suited to deal with stream processing graphs, and; (iii) there is a need to be able to schedule tasks with time-varying computational requirements efficiently. To address (i), we propose the heterogeneous value with load balancing and communication contention (HVLB_CC) (A) algorithm, which considers load balancing in addition to the parameters considered by the HSV_CC algorithm. We propose HVLB_CC (B) to address issue (ii). HVLB_CC (B) can deal with stream processing task graphs and more various directed acyclic graphs to prevent assigning a higher priority to successor tasks. In addition, to address issue (iii), we propose HVLB_CC_IC. To schedule tasks more efficiently with various computation times, HVLB_CC_IC utilizes schedule holes left in processors. These idle time slots can be used for the execution of an optional part to generate more precise data results by applying imprecise computation models. Experimental results demonstrate that the proposed algorithms improve minimum schedule length, accuracy, and load balancing significantly compared to the HSV_CC algorithm. In addition, the proposed HVLB_CC (B) algorithm can schedule more varied task graphs without reducing performance, and, using imprecise computation models, HVLB_CC_IC yields higher precision data than HVLB_CC without imprecise computation models., Comment: Accepted for publication at Journal of Parallel and Distributed Computing (Elsevier) 2017

Visual Data stream Monitor (VisDM) is a new approach to integrate a visual programming language with a data stream management system (DSMS) to support the construction, configuration, and visualization of data stream applications through a set of building blocks, Visual Data Flow Components (VDFCs). This functionality is provided by extending the LabVIEW visual programming platform to support easy declarative specification of continuous visualizations of continuous query results. With actor-based data flows, visualization of data stream output becomes more accessible.

International audience; As distributed IoT applications become larger and more complex, the simple processing of raw sensor and actuation data streams becomes impractical. Instead, data streams must be fused into tangible facts and these pieces of information must be combined with a background knowledge to infer new bits of knowledge. And since many IoT applications require almost real-time reactivity to stimuli from the environment this information inference process has to be performed in a continuous, on-line manner. This paper proposes a new semantic model for data stream processing and real-time symbolic reasoning based on the concepts of Semantic Stream and Fact Stream, as a natural extensions of Complex Event Processing (CEP) and RDF (graph-based knowledge model). The main advantages of our approach are that: (a) it considers time as a key relation between pieces of information; (b) the processing of streams can be implemented using CEP and that (c) it is general enough to be applied to any Data Stream Management System (DSMS).

The field of fault detection and diagnosis has been the subject of considerable interest in industry. Fault detection may increase the availability of products, thereby improving their quality. Fau ...

data stream management system is used to manage and query coming large, continuous, fast and flexible data stream. The system is based on the flow of data extraction, transformation, combination, which is the main content and task query execution. This paper mainly discusses the design and implementation of query execution module and query execution is composed of two parts which include query operations, query execution and scheduling.

Due to the great progress of computer technology and mature development of network, more and more data are generated and distributed through the network, which is called data streams. During the last couple of years, a number of researchers have paid their attention to data stream management, which is different from the conventional database management. At present, the new type of data management system, called data stream management system (DSMS), has become one of the most popular research areas in data engineering field. Lots of research projects have made great progress in this area. Since the current DSMS does not support queries on sequence data, this paper, we will focus on multi-attribute streams, such as video films. We will discuss the related issues such as how to build an efficient index for all queries of different streams and the corresponding query processing mechanisms.

Big Data applications require real-time processing of complex computations on streaming and static information. Applications such as the diagnosis of power generating turbines require the integration of high velocity streaming and large volume of static data from multiple sources. In this paper we study various optimisations related to efficiently processing of streaming and static information. We introduce novel indexing structures for stream processing, a query-planner component that decides when their creation is beneficial, and we examine precomputed summarisations on archived measurements to accelerate streaming and static information processing. To put our ideas into practise, we have developed Exa Stream, a data stream management system that is scalable, has declarative semantics, supports user defined functions, and allows efficient execution of complex analytical queries on streaming and static data. Our work is accompanied by an empirical evaluation of our optimisation techniques.

Video on demand (VOD) server requires a large storage capacity and rapid video retrieval from this storage to quickly stream these videos to users. The hybrid storage system, which combines HDDs and SSD in the server, has become popular because of such requirements. The combination of both storage modes is relatively weak in terms of optimising fast access prior and in supporting the production of a high number of simultaneous streams. This paper presents an enhanced hybrid storage system (EHSS)-based VOD server to improve the performance of the VOD server. The design of the EHSS and its streaming management scheme produce high performance and satisfy the performance requirements of a VOD server in terms of I/O throughput and access latency. The experimental results show that the proposed work provides better performance than the VOD server-based FADM because it enhances the average response times for the various scales of intensive workload by 69.89%.

Many emerging applications need continuous querying over uncertain event streams, mostly for online monitoring. These streaming uncertain events may come from radars, sensors, or even software hooks. The uncertainty is usually due to measurement errors, inherent ambiguities and privacy preserving reasons. To cover new requirements, we have designed and implemented a new system called Probabilistic Data Stream Management System (PDSMS) in Ref. 1. PDSMS is a data processing engine which runs continuous queries over probabilistic streams. However, lack of a semantics for probabilistic databases which supports continuous distributions prevented us from having a strong foundation for our query operators. It also precludes us from proving consistency and correctness of query operations especially after optimization and adaption. In fact, in the probabilistic database literature, there is no semantics available which covers continuous distributions. This limitation is very restrictive as in real-world, uncertainty is usually modeled by continuous distributions. In this paper, after presenting a basic probabilistic data model for PDSMS, we focus on querying and formally present the first semantics for probabilistic query operations which supports continuous distributions as well as discrete ones. Using this new semantics, we define our query operators (e.g. select, project, and join) formally without ambiguity and compatible with operators in relational algebra. Thus, we can leverage many transformation rules in relational algebra as well. This new semantics allows us to have different strictness levels and consistency between operators. We also proved many strictness theorems about different alternatives for query operators.