A Journey Towards Building Real-Time Big Data Analytics Environment for Drilling Operations: Challenges and Lessons Learned

Establishing a Big Data Analytics environment for Drilling operations is impacted by many factors such as the nature of drilling operations that requires quick decision making and has a diversity of data sources. This paper goes over the process of building a Real-Time Big Data Analytics environment and highlights key learning and challenges from this endeavor. The process of building such an environment is multifaceted. It involves building a Big Data solution architecture capable of handling the expected workload which includes servers, network, and software architecture. In addition, it requires establishing a data engineering pipeline that enables combining drilling data sets coming from different sources, reformatting and optimizing it to simplify and speed-up analytics. The data engineering pipeline is used to tackle different drilling data types which include a master structured database that describes the drilling planning, configurations, and operations, real-time drilling sensor data stored in WITSML semi-structured format (an E&P specific extension of XML), in addition to other unstructured content such as drilling reports. Data quality procedures were also added to the pipeline in order to identify and handle outliers in the data. In addition, data wrangling procedures were developed to ensure that the data is in the right shape and ready for analytics and machine learning. Moreover, this drilling analytics environment was built while maintaining all business security policies and fine-grained access control rules defined on the original data sources. The established Big Data Analytics environment results in a simpler and a shorter data science lifecycle and thus making it easy to combine, explore and deploy analytical models. This leads to more efficient business operations. Building a successful analytics environment requires much more than the technology piece. It involves many factors such as data ingestion, cleaning, wrangling while maintaining business security policies. This paper provide a unique perspective on how to build such an environment for real-time drilling operations.