1. Operator as a Service: Stateful Serverless Complex Event Processing
- Author
-
Luthra, Manisha, Hennig, Sebastian, Razavi, Kamran, Wang, Lin, Koldehofe, Boris, Wu, Xintao, Jermaine, Chris, Xiong, Li, Hu, Xiaohua Tony, Kotevska, Olivera, Lu, Siyuan, Xu, Weijia, Aluru, Srinivas, Zhai, Chengxiang, Al-Masri, Eyhab, Chen, Zhiyuan, Saltz, Jeff, Distributed Systems, Wu, Xintao, Jermaine, Chris, Xiong, Li, Hu, Xiaohua Tony, Kotevska, Olivera, Lu, Siyuan, Xu, Weijia, Aluru, Srinivas, Zhai, Chengxiang, Al-Masri, Eyhab, Chen, Zhiyuan, Saltz, Jeff, Computer Systems, and Network Institute
- Subjects
FOS: Computer and information sciences ,SDG 16 - Peace ,Computer science ,Distributed computing ,Data management ,Internet of Things ,Complex event processing ,02 engineering and technology ,Complex Event Processing ,Function as a Service ,Computer Science - Networking and Internet Architecture ,Runtime system ,Stateful firewall ,020204 information systems ,0202 electrical engineering, electronic engineering, information engineering ,Flexibility (engineering) ,Networking and Internet Architecture (cs.NI) ,business.industry ,SDG 16 - Peace, Justice and Strong Institutions ,Specification language ,Serverless computing ,Justice and Strong Institutions ,Computer Science - Distributed, Parallel, and Cluster Computing ,Scalability ,Key (cryptography) ,Distributed, Parallel, and Cluster Computing (cs.DC) ,business - Abstract
Complex Event Processing (CEP) is a powerful paradigm for scalable data management that is employed in many real-world scenarios such as detecting credit card fraud in banks. The so-called complex events are expressed using a specification language that is typically implemented and executed on a specific runtime system. While the tight coupling of these two components has been regarded as the key for supporting CEP at high performance, such dependencies pose several inherent challenges as follows. (1) Application development atop a CEP system requires extensive knowledge of how the runtime system operates, which is typically highly complex in nature. (2) The specification language dependence requires the need of domain experts and further restricts and steepens the learning curve for application developers. In this paper, we propose CEPLESS, a scalable data management system that decouples the specification from the runtime system by building on the principles of serverless computing. CEPLESS provides operator as a service and offers flexibility by enabling the development of CEP application in any specification language while abstracting away the complexity of the CEP runtime system. As part of CEPLESS, we designed and evaluated novel mechanisms for in-memory processing and batching that enables the stateful processing of CEP operators even under high rates of ingested events. Our evaluation demonstrates that CEPLESS can be easily integrated into existing CEP systems like Apache Flink while attaining similar throughput under a high scale of events (up to 100K events per second) and dynamic operator update in up to 238 ms., 10 pages, Published in the Proceedings of the IEEE International Conference on Big Data
- Published
- 2020