Search

Assessing real-time performance of Artificial Lift Pumps is a prevalent time-series problem to tackle for natural gas operators in Eastern Australia. Multiple physics, data-driven, and hybrid approaches have been investigated to analyse or predict pump performance. However, these methods present a challenge in running compute-heavy algorithms on streaming time-series data. As there is limited research on novel approaches to tackle multivariate time-series analytics for Artificial Lift systems, this paper introduces a human-in-the-loop approach, where petroleum engineers label clustered time-series data to aid in streaming analytics. We rely on our recently developed novel approach of converting streaming time-series data into heatmap images to assist with real-time pump performance analytics. During this study, we were able to automate the labelling of streaming time-series data, which helped petroleum and well surveillance engineers better manage Artificial Lift Pumps through machine learning supported exception-based surveillance. The streaming analytics system developed as part of this research used historical time-series data from three hundred and fifty-nine (359) coal seam gas wells. The developed method is currently used by two natural gas operators, where the operators can accurately detect ten (10) performance-related events and five (5) anomalous events. This paper serves a two-fold purpose; first, we describe a step-by-step methodology that readers can use to reproduce the clustering method for multivariate time-series data. Second, we demonstrate how a human-in-the-loop approach adds value to the proposed method and achieves real-world results. [ABSTRACT FROM AUTHOR]

This work studies the issues related to dynamic memory management in Data Stream Processing, an emerging paradigm enabling the real-time processing of live data streams. In this paper, we consider two streaming parallel patterns and we discuss different implementation variants related to how dynamic memory is managed. The results show that the standard mechanisms provided by modern C++ are not entirely adequate for maximizing the performance. Instead, the combined use of an efficient general purpose memory allocator, a custom allocator optimized for the pattern considered and a custom variant of the C++ shared pointer mechanism, provides a performance improvement up to 16% on the best case. [ABSTRACT FROM AUTHOR]

The large-scale analysis of tweets in real-time using supervised sentiment analysis depicts a unique opportunity for communication and audience research. Bringing together machine learning and streaming analytics approaches in a distributed environment might help scholars to obtain valuable data from Twitter in order to immediately classify messages depending on the context with no restrictions of time or storage, empowering cross-sectional, longitudinal and experimental designs with new inputs. Even when communication and audience researchers begin to use computational methods, most of them remain unfamiliar with distributed technologies to face big data challenges. This paper describes the implementation of parallelized machine learning methods in Apache Spark to predict sentiments in real-time tweets and explains how this process can be scaled up using academic or commercial distributed computing when personal computers do not support computations and storage. We discuss the limitation of these methods and their implications in communication, audience and media studies. [ABSTRACT FROM AUTHOR]

Many analytic applications require analyzing user interaction data. In particular, such data can be aggregated over a window to build user activity profiles. Clustering such aggregate profiles is useful for grouping together users with similar behaviors, so that common models could be built for them. In this paper, we present an approach for clustering profiles that are incrementally maintained over a stream of updates. Owing to the potentially large number of users and high rate of interactions, maintaining profile clusters can have high processing and memory resource requirements. To tackle this problem, we apply distributed stream processing. However, in the presence of distributed state, it is a major challenge to partition the profiles over nodes such that memory and computation balance is maintained, while keeping the clustering accuracy high. Furthermore, in order to adapt to potentially changing user interaction patterns, the partitioning of profiles to nodes should be continuously revised, yet one should minimize the migration of profiles so as not to disturb the online processing of updates. We develop a re-partitioning technique that achieves all these goals. To achieve this, we keep micro-cluster summaries at each node and periodically collect these summaries at a central node to perform re-partitioning. We use a greedy algorithm with novel affinity heuristics to revise the partitioning and update the routing tables without introducing a lengthy pause. We showcase the effectiveness of our approach using an application that clusters customers of a telecommunications company based on their aggregate calling profiles. [ABSTRACT FROM AUTHOR]

Exploring Internet of Things (IoT) data streams generated by smart cities means not only transforming data into better business decisions in a timely way but also generating long-term location intelligence for developing new forms of urban governance and organization policies. This paper proposes a new architecture based on the edge-fog-cloud continuum to analyze IoT data streams for delivering data-driven insights in a smart parking scenario. [ABSTRACT FROM AUTHOR]

Purpose: Enterprises are increasingly taking actionable steps to transform existing business models through digital technologies for service transformation such as big data analytics (BDA). BDA capabilities offer financial institutions to source financial data, analyse data, insight and store such data and information on collaborative platforms for a quick decision-making process. Accordingly, this study identifies how BDA capabilities can be deployed to provide significant improvement for financial services agility. Design/methodology/approach: The study relied on survey data from 485 banking professionals' perspectives with BDA usage, IT capability development and financial service agility. The PLS-SEM technique was used to evaluate the underlying relationship and the applicability of the research framework proposed. Findings: Based on the empirical test from this study, distinctive BDA usage grounded on the concept of IT capability viewpoint proof that financial service agility could be enhanced provided enterprises develop technical capabilities alongside other relevant resources. Practical implications: The study further highlights the need for financial service managers to identify BDA technologies such as data mining, query and reporting, data visualisation, predictive modelling, streaming analytics, video analytics and voice analytics to focus on financial knowledge gathering and market observation. Financial managers can also deploy BDA tools to develop a strategic road map for data management, data transferability and knowledge discovery for customised financial products. Originality/value: This study is a useful contribution to the burgeoning discussion with emerging technologies such as BDA implication to improving enterprises operations. [ABSTRACT FROM AUTHOR]

Wavelet neural networks (WNN) have been applied in many fields to solve regression as well as classification problems. After the advent of big data, as data gets generated at a brisk pace, it is imperative to analyze it as soon as it is generated owing to the fact that the nature of the data may change dramatically in short time intervals. This is necessitated by the fact that big data is all pervasive and throws computational challenges for data scientists. Therefore, in this paper, we built an efficient Scalable, Parallelized Wavelet Neural Network (SPWNN) which employs the parallel stochastic gradient algorithm (SGD) algorithm. SPWNN is designed and developed under both static and streaming environments in the horizontal parallelization framework. SPWNN is implemented by using Morlet and Gaussian functions as activation functions. This study is conducted on big datasets like gas sensor data which has more than 4 million samples and medical research data which has more than 10,000 features, which are high dimensional in nature. The experimental analysis indicates that in the static environment, SPWNN with Morlet activation function outperformed SPWNN with Gaussian on the classification datasets. However, in the case of regression, there is no clear trend was observed. In contrast, in the streaming environment i.e., Gaussian outperformed Morlet on the classification and Morlet outperformed Gaussian on the regression datasets. Overall, the proposed SPWNN architecture achieved a speedup of 1.22 - 1.78. [ABSTRACT FROM AUTHOR]

Objectives: The purpose of this study was to Identify whether changes in heart rate variability (HRV) could be detected as critical illness resolves by comparing HRV from the time of pediatric intensive care unit (PICU) admission with HRV immediately prior to discharge. We also sought to demonstrate that HRV derived from electrocardiogram (ECG) data from bedside monitors can be calculated in critically-ill children using a real-time, streaming analytics platform. Methods: This was a retrospective, observational pilot study of 17 children aged 0 to 18 years admitted to the PICU of a free-standing, academic children’s hospital. Three time-domain measures of HRV were calculated in real-time from bedside monitor ECG data and stored for analysis. Measures included: root mean square of successive differences between NN intervals (RMSSD), percent of successive NN interval differences above 50 ms (pNN50), and the standard deviation of NN intervals (SDNN). Results: HRV values calculated from the first and last 24 hours of PICU stay were analyzed. Mixed effects models demonstrated that all three measures of HRV were significantly lower during the first 24 hours compared to the last 24 hours of PICU admission (p<0.001 for all three measures). In models exploring the relationship between time from admission and log HRV values, the predicted average HRV remained consistently higher in the last 24 hours of PICU stay compared to the first 24 hours. Conclusion: HRV was significantly lower in the first 24 hours compared to the 24 hours preceding PICU discharge, after resolution of critical illness. This demonstrates that it is feasible to detect changes in HRV using an automated, streaming analytics platform. Continuous tracking of HRV may serve as a marker of recovery in critically ill children. [ABSTRACT FROM AUTHOR]

We are seeing an enormous increase in the availability of streaming, time-series data. Largely driven by the rise of connected real-time data sources, this data presents technical challenges and opportunities. One fundamental capability for streaming analytics is to model each stream in an unsupervised fashion and detect unusual, anomalous behaviors in real-time. Early anomaly detection is valuable, yet it can be difficult to execute reliably in practice. Application constraints require systems to process data in real-time, not batches. Streaming data inherently exhibits concept drift, favoring algorithms that learn continuously. Furthermore, the massive number of independent streams in practice requires that anomaly detectors be fully automated. In this paper we propose a novel anomaly detection algorithm that meets these constraints. The technique is based on an online sequence memory algorithm called Hierarchical Temporal Memory (HTM). We also present results using the Numenta Anomaly Benchmark (NAB), a benchmark containing real-world data streams with labeled anomalies. The benchmark, the first of its kind, provides a controlled open-source environment for testing anomaly detection algorithms on streaming data. We present results and analysis for a wide range of algorithms on this benchmark, and discuss future challenges for the emerging field of streaming analytics. [ABSTRACT FROM AUTHOR]

By modeling double exponential smoothing as a weighted directed acyclic graph, we design an implementation of rolling window double exponential smoothing which is incremental-decremental in the sense that points can be added to and removed from the window with overhead and computation independent of the window size. This has applications to real-time streaming analytics systems having certain universality and flexibility requirements. [ABSTRACT FROM AUTHOR]

The Australian Commission for Safety and Quality in Health Care has created the National Safety and Quality Health Service standards that all hospitals must address in order to remain accredited. This case study details the first known digitisation of the 10 national quality and safety standards mandated in a quaternary integrated digital hospital. A team of clinical informaticians, information technology experts and clinicians was assembled. Data were chosen and the data were then extracted and validated and presented (often in near real time) in an easily consumable dashboard format with appropriate governance to allow clinicians and executives to monitor the quality and safety standards across the hospital. All 10 standards were defined and extracted contemporaneously from the digital hospital for every patient, every time. This is in stark contrast with traditional retrospective point prevalence surveys. This case study details the first known fully digital accreditation in a sophisticated integrated digital hospital. Digitisation of hospital quality and safety to produce real-time data is the future of clinical redesign to improve patient care. What is known about the topic?: Healthcare delivery is complex and the ability of healthcare providers to maintain consistent standards of quality and safety is variable. Traditionally, these standards have been assessed by intermittent retrospective point-prevalence survey activity. Sophisticated digital hospitals provide the opportunity to develop data and analytics that monitor quality and safety standards across every patient, every time in near real time. What does this paper add?: This paper describes a digital hospital which has created streaming analytics to monitor live performance of quality and safety standards. The necessary skills, leadership and governance for this process are outlined and the products described. What are the implications for practitioners?: Shifting from retrospective paper-based point prevalence surveys to a digital platform has several implications. Firstly, it is an imperative to drive digital transformation of Australian hospitals. Secondly, it provides data for intervention to the hospital staff, so that issues can be addressed and improved in real-time, rather than waiting for survey results. Lastly, this new model of maintaining quality and safety also requires the development of new skills in the hospital setting including data literacy, digital clinical governance and clinical informatics. [ABSTRACT FROM AUTHOR]

Nowadays, modern Big Stream Processing Solutions (e.g. Spark, Flink) are working towards being the ultimate framework for streaming analytics. In order to achieve this goal, they started to offer extensions of SQL that incorporate stream-oriented primitives such as windowing and Complex Event Processing (CEP). The former enables stateful computation on infinite sequences of data items while the latter focuses on the detection of events pattern. In most of the cases, data items and events are considered instantaneous, i.e., they are single time points in a discrete temporal domain. Nevertheless, a point-based time semantics does not satisfy the requirements of a number of use-cases. For instance, it is not possible to detect the interval during which the temperature increases until the temperature begins to decrease, nor for all the relations this interval subsumes. To tackle this challenge, we present D2IA; a set of novel abstract operators to define analytics on user-defined event intervals based on raw events and to efficiently reason about temporal relationships between intervals and/or point events. We realize the implementation of the concepts of D2IA on top of Flink, a distributed stream processing engine for big data. • A family of operators to generate interval events from instantaneous events. • Operators are defined as DSL to expressively generate data-driven intervals. • Operators fill a gap in the expressiveness of large-scale stream processing engines. • Two alternative implementations based on Apache Flink and Esper. • A systematic perfromance evaluation using the linear road benchmark. [ABSTRACT FROM AUTHOR]

In this prospective, interventional, international study, we investigate continuous monitoring of hospitalised patients' vital signs using wearable technology as a basis for real-time early warning scores (EWS) estimation and vital signs time-series prediction. The collected continuous monitored vital signs are heart rate, blood pressure, respiration rate, and oxygen saturation of a heterogeneous patient population hospitalised in cardiology, postsurgical, and dialysis wards. Two aspects are elaborated in this study. The first is the high-rate (every minute) estimation of the statistical values (e.g., minimum and mean) of the vital signs components of the EWS for one-minute segments in contrast with the conventional routine of 2 to 3 times per day. The second aspect explores the use of a hybrid machine learning algorithm of kNN-LS-SVM for predicting future values of monitored vital signs. It is demonstrated that a real-time implementation of EWS in clinical practice is possible. Furthermore, we showed a promising prediction performance of vital signs compared to the most recent state of the art of a boosted approach of LSTM. The reported mean absolute percentage errors of predicting one-hour averaged heart rate are 4.1, 4.5, and 5% for the upcoming one, two, and three hours respectively for cardiology patients. The obtained results in this study show the potential of using wearable technology to continuously monitor the vital signs of hospitalised patients as the real-time estimation of EWS in addition to a reliable prediction of the future values of these vital signs is presented. Ultimately, both approaches of high-rate EWS computation and vital signs time-series prediction is promising to provide efficient cost-utility, ease of mobility and portability, streaming analytics, and early warning for vital signs deterioration. [ABSTRACT FROM AUTHOR]

In this work, we focus on distance-based outliers in a metric space, where the status of an entity as to whether it is an outlier is based on the number of other entities in its neighborhood. In recent years, several solutions have tackled the problem of distance-based outliers in data streams, where outliers must be mined continuously as new elements become available. An interesting research problem is to combine the streaming environment with massively parallel systems to provide scalable stream-based algorithms. However, none of the previously proposed techniques refer to a massively parallel setting. Our proposal fills this gap and investigates the challenges in transferring state-of-the-art techniques to Apache Flink, a modern platform for intensive streaming analytics. We thoroughly present the technical challenges encountered and the alternatives that may be applied, of which a micro-clustering-based one is the most efficient. We show speed-ups of up to 2.27 times over advanced non-parallel solutions, by using just an ordinary four-core machine and a real-world dataset. When moving to a three-machine cluster, due to less contention, we manage to achieve both better scalability in terms of the window slide size and the data dimensionality, and even higher speed-ups, e.g., by a factor of more than 11X. Overall, our results demonstrate that outlier mining can be achieved in an efficient and scalable manner. The resulting techniques have been made publicly available as open-source software. • We explore a series of implementation alternatives, differing in the algorithmic features they employ. • We provide thorough experimental evaluation results. • Overall, we demonstrate how outlier mining can be achieved in an efficient and scalable manner through parallelism. • We offer the source code as an open-source library. [ABSTRACT FROM AUTHOR]

The expansion of the electronic commerce, together with an increasing confidence of customers in electronic payments, makes of fraud detection a critical factor. Detecting frauds in (nearly) real time setting demands the design and the implementation of scalable learning techniques able to ingest and analyse massive amounts of streaming data. Recent advances in analytics and the availability of open source solutions for Big Data storage and processing open new perspectives to the fraud detection field. In this paper we present a Scalable Real-time Fraud Finder (SCARFF) which integrates Big Data tools (Kafka, Spark and Cassandra) with a machine learning approach which deals with imbalance, nonstationarity and feedback latency. Experimental results on a massive dataset of real credit card transactions show that this framework is scalable, efficient and accurate over a big stream of transactions. [ABSTRACT FROM AUTHOR]

Extracting latent low-dimensional structure from high-dimensional data is of paramount importance in timely inference tasks encountered with “Big Data” analytics. However, increasingly noisy, heterogeneous, and incomplete datasets, as well as the need for real-time processing of streaming data, pose major challenges to this end. In this context, the present paper permeates benefits from rank minimization to scalable imputation of missing data, via tracking low-dimensional subspaces and unraveling latent (possibly multi-way) structure from incomplete streaming data. For low-rank matrix data, a subspace estimator is proposed based on an exponentially weighted least-squares criterion regularized with the nuclear norm. After recasting the nonseparable nuclear norm into a form amenable to online optimization, real-time algorithms with complementary strengths are developed, and their convergence is established under simplifying technical assumptions. In a stationary setting, the asymptotic estimates obtained offer the well-documented performance guarantees of the batch nuclear-norm regularized estimator. Under the same unifying framework, a novel online (adaptive) algorithm is developed to obtain multi-way decompositions of low-rank tensors with missing entries and perform imputation as a byproduct. Simulated tests with both synthetic as well as real Internet and cardiac magnetic resonance imagery (MRI) data confirm the efficacy of the proposed algorithms, and their superior performance relative to state-of-the-art alternatives. [ABSTRACT FROM PUBLISHER]