Search

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

With the increasing complexity of IT outsourcing environments thousands of servers and their configurations are increasingly managed by globally distributed teams. This requires a flexible identity access management process in place to efficiently provision necessary access rights for a given system, only if users need it, when they need it and for only as long as they need it. In this paper we present a novel approach to discovering required role permissions by integrating system data and enterprise crowd sourcing (a process where a group of experts solve problems through collaboration). By mining server registries, compliance repositories (such as user revalidation records), we derive a set of servers and the respective access rights for each team member. This data is then validated and updated by one or more team members using the principles of crowd sourcing. We show that this approach improves the role discovery process and accelerates the deployment of the security service infrastructure.