VectorH: taking SQL-on-Hadoop to the next level

In this paper we describe VectorH: a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and scalable data storage by relying on HDFS, extending the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS block replication policy to ensure local reads under most circumstances. VectorH integrates with YARN for workload management, achieving a high degree of elasticity . Even though HDFS is an append-only filesystem, and it supports ordered table storage, VectorH can accommodate trickle updates through Positional Delta Trees (PDTs), a differential update structure that can be queried efficiently. We describe the main technical extensions to single-server Vectorwise that turned it into a Hadoop-based MPP system, in terms of workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. In the evaluation section we compare VectorH with HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance than these competitors.