Structure aware resource estimation for effective scheduling and execution of data intensive workflows in cloud

A set of interdependent tasks used to automate a business or scientific process can be modelled as a workflow and represented in the form of a Directed Acyclic Graph (DAG) or Directed Acyclic Graph in XML (DAX). Cloud computing is the current popular technology that provides hardware and software resources that are accessible from anywhere and at any time. As the cloud users are relieved of the difficulties of managing hardware and software resources, it is the most convenient and suitable environment to execute workflows. Workflows that accept and process a large amount of data are termed as data intensive workflows. The execution cost of such workflows in the cloud depends not only on the configuration of the Virtual Machines (VMs) but also the cost of data transfer between the tasks. Due to the highly dynamic arrangement of tasks in the workflow, deciding the optimum configuration and exact number of VMs is a big challenge for researchers today. Hence, in this paper, an effective resource provisioning and scheduling mechanism based on the structure of the workflow is proposed. The significance of this work is to identify the required number of VMs and their configuration, based on the structure of the workflow and optimizing data transfer between the tasks. Popular workflows like Montage, CyberShake, Epigenomics and Inspiral are used to analyse the quality of this work, and the obtained results confirm that the proposed workflow scheduler is able to provide a notable reduction in execution cost without compromising the execution time.