Search

Today, the emergence of the industry revolution systems such as Industry 4.0, Internet of Things, and big data frameworks poses new challenges in terms of storage and processing of real-time data. As systems scale in humongous sizes, a crucial task is to administer the variety of different subsystems and applications to ensure high performance. This is directly related with the identification and elimination of system failures and errors, while the system runs. In particular, database systems may experience abnormalities related with decreased throughput or increased resource usage, that in turn affects system performance. In this article, we focus on not only SQL (NoSQL) database systems that are ideal for storing sensor data in the concept of Industry 4.0. This typically includes a variety of applications and workloads that are difficult to online monitor, thus making anomaly detection a challenging task. Creating a robust platform to serve such infrastructures with minimum hardware or software failures is a key challenge. In this article, we propose RADAR, an anomaly detection system that works on real time. RADAR is a data-driven decision-making system for NoSQL systems, by providing process information extraction during resource monitoring and by associating resource usage with the top processes, to identify anomalous cases. In this article, we focus on anomalies such as hardware failures or software bugs that could lead to abnormal application runs, without necessarily stopping system functionality, e.g., due to a system crash, but by affecting its performance, e.g., decreased database system throughput. Although different patterns may occur through time, we focus on periodic running workloads (e.g., monitoring daily usage) that are very common for NoSQL systems, and Internet of Things scenarios where data streams are forwarded to the Cloud for storage and processing. We apply various machine learning algorithms such as autoregressive integrated moving average (ARIMA), seasonal ARIMA, and long–short-term memory recurrent neural networks. We experimentally analyze our solution to demonstrate the benefits of supporting online erroneous state identification and characterization for modern applications.

Cloud computing has emerged as a new paradigm that offers on-demand availability and flexible pricing models. However, cloud applications are being transformed into large scale systems where managing and monitoring cloud resources becomes a challenging task. System administrators are in need of automated tools to effectively detect abnormal system behaviour and ensure the Service Level Agreement (SLA) between the service user and the service provider. In this work, we propose a framework for online anomaly detection based on cloud application metrics. We utilize Recurrent Neural Networks for learning normal sequence representations and predict future events. Then, we use the predicted sequence as the representative sequence of normal events and based on the Dynamic Time Warping algorithm we classify future time series as normal or abnormal. Furthermore, to create a real world scenario and validate the proposed method, we used Yahoo! Cloud Serving Benchmark as a state-of-the-art benchmark tool for cloud data serving systems. Our experimental analysis shows the ability of the proposed approach to detect abnormal behaviours of NoSQL systems on-the-fly with minimum instrumentation.

Cloud computing technology is on the rise as it provides an easy to scale environment for Internet users in terms of computational resources. At the same time, cloud providers manage this demand for computational power by offering a pay per use model for virtualized resources. Yet, it is a challenging issue to administer the variety of different cloud applications and ensure high performance by identifying failures and errors on runtime. Distributed applications are error-prone, and creating a platform to support minimum hardware and software failures is a key challenge. In this work, we focus on anomaly detection of data storage systems, and we propose a solution for detecting performance degradation of cloud deployed systems in real time. We use Long Short-Term Memory (LSTM) Autoencoders for learning the normal representations and reconstruct the input sequences. Then, we used the reconstructed errors of the LSTM Autoencoders on unseen time series data to detect abnormal behaviours. We used state-of-the-art benchmarks such as TPCx-IoT and YCSB to evaluate the performance of HBase and MongoDB systems. Our experimental analysis shows the ability of the proposed approach to detect abnormal behaviours in cloud systems.