Search

The complexity of many problems in science and engineering requires computational capacity exceeding what the average user can expect from a single computational center. While many of these problems can be viewed as a set of independent tasks, their collective complexity easily requires millions of core-hours on any high-power computing (HPC) resource, and throughput that can't be sustained by a single, multiuser queuing system. An exploration of the use of aggregated HPC resources to solve large-scale engineering problems shows that it's possible to build a computational federation that's easy for end users to implement, and is elastic, resilient, and scalable. Here, the authors argue that the fusion of federated computing and real-life engineering problems can be brought to the average user if relevant middleware is provided. They report on the use of federation of 10 distributed heterogeneous HPC resources to perform a large-scale interrogation of the parameter space in the microscale fluid flow problem.

Clouds are rapidly joining high-performance computing (HPC) systems, clusters, and grids as viable platforms for scientific exploration and discovery. As a result, understanding application formulations and usage modes that are meaningful in such a hybrid infrastructure, and how application workflows can effectively utilize it, is critical. Here, three hybrid HPC/grid and cloud cyber infrastructure usage modes are explored: HPC in the Cloud, HPC plus Cloud, and HPC as a Service, presenting illustrative scenarios in each case and outlining benefits, limitations, and research challenges.

The goal of Grid computing is to integrate the usage of computer resources from cooperating partners in the form of Virtual Organizations (VO). One of its key functions is to match jobs to execution resources efficiently. For interoperability between VOs, this matching operation occurs in resource brokering middleware, commonly referred to as the meta-scheduler or meta-broker. In this paper, we present an approach to a meta-scheduler architecture, combining hierarchical and peer-to-peer models for flexibility and extensibility. Interoperability is further promoted through the introduction of a set of protocols, allowing meta-schedulers to maintain sessions and exchange job and resource state using Web Services. Our architecture also incorporates a resource model that enables an efficient resource matching across multiple Virtual Organizations, especially where the compute resources and state are dynamic. Experiments demonstrate these new functional features across three distributed organizations (BSC, FIU, and IBM), that internally use different job scheduling technologies, computing infrastructure and security mechanisms. Performance evaluations through actual system measurements and simulations provide the insights on the architecture's effectiveness and scalability.

Social Clouds provide the capability to share resources among participants within a social network-leveraging on the trust relationships already existing between such participants. In such a system, users are able to trade resources between each other rather than make use of capability offered at a centralized data center. Although such an environment has significant potential for improving resource utilization and making available additional capacity that remains dormant, incentives for sharing remain an important hurdle limiting its effective. In this paper, we utilize the socioeconomic model proposed by Silvio Gesell to demonstrate how a 'virtual currency' can be used to incentivise sharing of resources within a 'community'. We subsequently demonstrate, through simulations, the benefit provided to participants within such a community using a variety of economic such as overall credits gained and technical number of successfully completed transactions metrics. Further, we describe our implementation of such a Social Cloud using CometCloud. CometCloud is an autonomic computing engine for cloud and grid environments. It supports highly heterogeneous and dynamic federated cloud/Grid infrastructures, integration of public/private clouds and autonomic cloudbursts. We demonstrate the implementation of two designs on the basis of the master/worker approach: i one tuple space per cluster and ii one coordination tuple space and multiple transient spaces-one per each cluster. Finally, we discuss an extended version of our Social Cloud model where intermediary relay nodes take on more active roles as traders in a transaction. Copyright © 2013 John Wiley & Sons, Ltd.

Scientific simulation workflows executing on very large scale computing systems are essential modalities for scientific investigation. The increasing scales and resolution of these simulations provide new opportunities for accurately modeling complex natural and engineered phenomena. However, the increasing complexity necessitates managing, transporting, and processing unprecedented amounts of data, and as a result, researchers are increasingly exploring data-staging and in-situ workflows to reduce data movement and data-related overheads. However, as these workflows become more dynamic in their structures and behaviors, data staging and in-situ solutions must evolve to support new requirements.In this paper, we explore how the service-oriented concept can be applied to extreme-scale in-situ workflows. Specifically, we explore persistent data staging as a service and present the design and implementation of DataSpaces as a Service, a service-oriented data staging framework. We use a dynamically coupled fusion simulation workflow to illustrate the capabilities of this framework and evaluate its performance and scalability.

The increasing gap between available compute power and I/O capabilities is resulting in simulation pipelines running on leadership computing facilities being reformulated. In particular, in-situ processing is complementing conventional post-process analysis, however, it can be performed by using the same compute resources as the simulation or using secondary dedicated resources. In this paper, we focus on three different in-situ analysis strategies, which use the same compute resources as the ongoing simulation but different data movement strategies. We evaluate the costs incurred by these strategies in terms of run time, scalability and power/energy consumption. Furthermore, we extrapolate power behavior to peta-scale and investigate different design choices through projections. Experimental evaluation at full machine scale on Titan supports that using fewer cores per node for in-situ analysis is the optimum choice in terms of scalability. Hence, further research effort should be devoted towards developing in-situ analysis techniques following this strategy in future high-end systems.

This chapter contains sections titled: Introduction Background and Related Work Proactive, Component-Based Power Management Quantifying Energy Saving Possibilities Evaluation of the Proposed Strategies Results Concluding Remarks Summary References ]]>

Virtualized datacenters and clouds are being increasingly considered for traditional High-Performance Computing (HPC) workloads that have typically targeted Grids and conventional HPC platforms. However, maximizing energy efficiency and utilization of datacenter resources, and minimizing undesired thermal behavior while ensuring application performance and other Quality of Service (QoS) guarantees for HPC applications requires careful consideration of important and extremely challenging tradeoffs. Virtual Machine (VM) migration is one of the most common techniques used to alleviate thermal anomalies (i.e., hotspots) in cloud datacenter servers as it reduces load and, hence, the server utilization. In this article, the benefits of using other techniques such as voltage scaling and pinning (traditionally used for reducing energy consumption) for thermal management over VM migrations are studied in detail. As no single technique is the most efficient to meet temperature/performance optimization goals in all situations, an autonomic approach that performs energy-efficient thermal management while ensuring the QoS delivered to the users is proposed. To address the problem of VM allocation that arises during VM migrations, an innovative application-centric energy-aware strategy for Virtual Machine (VM) allocation is proposed. The proposed strategy ensures high resource utilization and energy efficiency through VM consolidation while satisfying application QoS by exploiting knowledge obtained through application profiling along multiple dimensions (CPU, memory, and network bandwidth utilization). To support our arguments, we present the results obtained from an experimental evaluation on real hardware using HPC workloads under different scenarios.

In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.

The increasing demand for high performance computing resources has led to new forms of collaboration of distributed systems, such as grid computing systems. Moreover, the need for interoperability among different grid systems through the use of common protocols and standards has become increased in the last few years. In this paper we describe and evaluate scheduling techniques for multiple grid scenarios. In particular, they consist of the proposed ''bestBrokerRank'' broker selection policy and two different variants. The first one uses the resource information in aggregated forms as input, and the second one also uses the broker average bounded slowdown as a dynamic performance metric. From our evaluations performed with simulation tools, we state that, although the aggregation algorithms lose resource information accuracy, the broker selection policies using aggregated resource data do not penalize their performance significantly. Moreover, we show that the best performance results are obtained with the coordinated policy using dynamic performance information, in addition to aggregated resource information. Therefore, we conclude that delegating part of the scheduling responsibilities to the underlying scheduling layers promotes separation of concerns and is a good way to balance the performance among the different grid systems.

In this paper, we investigate real-world scenarios in which MapReduce programming model and specifically Hadoop framework could be used for processing large-scale, geographically scattered datasets. We propose an Adaptive Reduce Task Scheduling (ARTS) algorithm and evaluate it on a distributed Hadoop cluster involving multiple datacenters as well as the on a shared Hadoop cluster. The evaluation demonstrates that the ARTS algorithm outperforms the default Reduce phase scheduling algorithm in Hadoop framework.

Sensitivity analysis (SA) is a fundamental tool of uncertainty quantification(UQ). Adjoint-based SA is the optimal approach in many large-scale applications, such as the direct numerical simulation (DNS) of combustion. However, one of the challenges of the adjoint workflow for time-dependent applications is the storage and I/O requirements for the application state. During the time-reversal portion of the workflow, forward state is required in last-in-first-out order. The resulting requirements for storage at exascale are enormous. To mitigate this requirement, application state is regenerated from checkpoints over short windows of application time. This approach drastically reduces the total volume of stored data, allows the caching of state in the regeneration window in memory and on local SSDs, may accelerate the application execution by reducing output frequency, and reduces the power overhead from I/O. We explore variations to this workflow, applied to a proxy for the SA of turbulent combustion, by varying checkpoint number, state storage, and other regeneration options to find efficient implementations for minimizing compute time or power consumption.

Global energy problems necessitate an urgent transformation of the existing electrical generation grid into a smart grid, rather than a gradual evolution. A smart grid is a real-time bi-directional communication network between end users and their utility companies which monitors power demand and manages the provisioning and transport of electricity from all generation sources. As a crucial part of this transformation, increasing numbers of smart meters generate correspondingly increasing amounts of data every day. Analyzing this data to extract insight into, and to maintain control over energy usage has become a big data problem - one which cannot be handled manually, and which requires autonomic computing solutions. In this paper, we examine electric vehicles (EVs) as a use case to investigate how to use social media, sensing data, and big data analytics to optimize smart grid management. We discuss the requirements to realize such an approach and describe an autonomic system architecture and a possible design. We believe the proposed architecture and strategy will help optimize how provisioning is performed in a smart grid, even when smart meters are not available.

Cloud computing technology is being adopted by a growing number of organizations as a way to manage, store and process data. Protecting this infrastructure and key resources such as data and recovering against cyber threats has become a critical concern. At the same time, cloud computing technology such as virtualization and on-demand provisioning can be leveraged to implement protection and recovery mechanisms for sensitive computing services. However, the energy costs of such self-healing mechanisms has been ignored. In this paper, we present a novel IDS-based self-protection mechanism at the virtual machine level inspired by software rejuvenation concepts, and a proof-of-concept prototype built upon state-of-the-art cloud technology. Our empirical evaluation aims at characterizing the cost of the proposed strategy in terms of response time and energy consumption. Preliminary results show a clear correlation between IDS accuracy, attack rate, cloud system workload, energy consumption, and response time.

Background The development of digital imaging technology is creating extraordinary levels of accuracy that provide support for improved reliability in different aspects of the image analysis, such as content-based image retrieval, image segmentation, and classification. This has dramatically increased the volume and rate at which data are generated. Together these facts make querying and sharing non-trivial and render centralized solutions unfeasible. Moreover, in many cases this data is often distributed and must be shared across multiple institutions requiring decentralized solutions. In this context, a new generation of data/information driven applications must be developed to take advantage of the national advanced cyber-infrastructure (ACI) which enable investigators to seamlessly and securely interact with information/data which is distributed across geographically disparate resources. This paper presents the development and evaluation of a novel content-based image retrieval (CBIR) framework. The methods were tested extensively using both peripheral blood smears and renal glomeruli specimens. The datasets and performance were evaluated by two pathologists to determine the concordance. Results The CBIR algorithms that were developed can reliably retrieve the candidate image patches exhibiting intensity and morphological characteristics that are most similar to a given query image. The methods described in this paper are able to reliably discriminate among subtle staining differences and spatial pattern distributions. By integrating a newly developed dual-similarity relevance feedback module into the CBIR framework, the CBIR results were improved substantially. By aggregating the computational power of high performance computing (HPC) and cloud resources, we demonstrated that the method can be successfully executed in minutes on the Cloud compared to weeks using standard computers. Conclusions In this paper, we present a set of newly developed CBIR algorithms and validate them using two different pathology applications, which are regularly evaluated in the practice of pathology. Comparative experimental results demonstrate excellent performance throughout the course of a set of systematic studies. Additionally, we present and evaluate a framework to enable the execution of these algorithms across distributed resources. We show how parallel searching of content-wise similar images in the dataset significantly reduces the overall computational time to ensure the practical utility of the proposed CBIR algorithms.

The development of digital imaging technology is creating extraordinary levels of accuracy that provide support for improved reliability in different aspects of the image analysis such as content-based image retrieval, image segmentation and classification. This has dramatically increased the volume and generation rates of data, which make querying and sharing non-trivial, and render centralized solutions infeasible. Moreover, in many cases this data is naturally distributed or has to be shared across multiple institutions requiring decentralized solutions. In this paper, we present a federation approach that can take advantage of advanced cyber-infrastructure (ACI) by seamlessly and securely interacting with information/data located across geographically distributed resources. We describe and evaluate the establishment of a federated marketplace across resources from UK and US in the context of collaborative research in medical imaging. We show how users from different sites can transparently access to remote data by placing computational requests in the marketplace. Moreover, we evaluate different pricing policies to demonstrate how site providers and users can benefit from such federation.

Cloud computing has emerged as a dominant paradigm that has been widely adopted by enterprises. Clouds provide on-demand access to computing utilities, an abstraction of unlimited computing resources, and support for on-demand scale up, scale down and scale out. Clouds are also rapidly joining high performance computing system, clusters and grids as viable platforms for scientific exploration and discovery. Furthermore, dynamically federated Cloud-of-Clouds infrastructure can support heterogeneous and highly dynamic applications requirements by composing appropriate (public and/or private) cloud services and capabilities. As a result, providing scalable and robust mechanisms to federate distributed infrastructures and handle application workflows, that can effectively utilize them, is critical. In this chapter, we present a federation model to support the dynamic federation of resources and autonomic management mechanisms that coordinate multiple workflows to use resources based on objectives. We demonstrate the effectiveness of the proposed framework and autonomic mechanisms through the discussion of an experimental evaluation of illustrative use case application scenarios, and from these experiences, we discuss that such a federation model can support new types of application formulations.

The increasing gap between the rate at which large scale scientific simulations generate data and the corresponding storage speeds and capacities is leading to more complex system architectures with deep memory hierarchies. Advances in non-volatile memory (NVRAM) technology have made it an attractive candidate as intermediate storage in this memory hierarchy to address the latency and performance gap between main memory and disk storage. As a result, it is important to understand and model its energy/performance behavior from an application perspective as well as how it can be effectively used for staging data within an application workflow. In this paper, we target a NVRAM-based deep memory hierarchy and explore its potential for supporting in-situ/in-transit data analytics pipelines that are part of application workflows patterns. Specifically, we model the memory hierarchy and experimentally explore energy/performance behaviors of different data management strategies and data exchange patterns, as well as the tradeoffs associated with data placement, data movement and data processing.

We present a federation model to support the dynamic federation of resources and autonomic management mechanisms that coordinate multiple workflows to use resources based on objectives. We illustrate the effectiveness of the proposed framework and autonomic mechanisms through the discussion of representative use case application scenarios, and from these experiences, we discuss that such a federation model can support new types of application formulations.

Emerging scientific simulations on leadership class systems are generating huge amounts of data. However, the increasing gap between computation and disk I/O speeds makes traditional data analytics pipelines based on post-processing cost prohibitive and often infeasible. In this paper, we investigate an alternate approach that aims to bring the analytics closer to the data using data staging and the in-situ execution of data analysis operations. Specifically, we present the design, implementation and evaluation of a framework that can support in-situ feature-based object tracking on distributed scientific datasets. Central to this framework is the scalable decentralized and online clustering (DOC) and cluster tracking algorithm, which executes in-situ (on different cores) and in parallel with the simulation processes, and retrieves data from the simulations directly via on-chip shared memory. The results from our experimental evaluation demonstrate that the in-situ approach significantly reduces the cost of data movement, that the presented framework can support scalable feature-based object tracking, and that it can be effectively used for in-situ analytics for large scale simulations.

The purpose of this chapter is to identify and analyze the challenges of creating new software in the public cloud due to legal regulations. Specifically, this chapter explores how the Sarbanes-Oxley Act (SOX) will indirectly affect the development and implementation process of cloud computing applications in terms of software engineering and actual legality of said software solutions. The goal of this chapter is twofold - to bring attention to the need for specific analysis of legal issues in public clouds (as opposed to general analysis), and to illustrate the need for cloud developers to address legal constraint while creating their platforms, in order to increase their viability in the corporate environment.

High-performance parallel computing architectures are increasingly based on multi-core processors. While current commercially available processors are at 8 and 16 cores, technological and power constraints are limiting the performance growth of the cores and are resulting in architectures with much higher core counts, such as the experimental many-core Intel Single-chip Cloud Computer (SCC) platform. These trends are presenting new sets of challenges to HPC applications including programming complexity and the need for extreme energy efficiency.In this paper, we first investigate the power behavior of scientific Partitioned Global Address Space (PGAS) application kernels on the SCC platform, and explore opportunities and challenges for power management within the PGAS framework. Results obtained via empirical evaluation of Unified Parallel C (UPC) applications on the SCC platform under different constraints, show that, for specific operations, the potential for energy savings in PGAS is large; and power/performance trade-offs can be effectively managed using a cross-layer approach. We investigate cross-layer power management using PGAS language extensions and runtime mechanisms that manipulate power/performance tradeoffs. Specifically, we present the design, implementation and evaluation of such a middleware for application-aware cross-layer power management of UPC applications on the SCC platform. Finally, based on our observations, we provide a set of insights that can be used to support similar power management for PGAS applications on other many-core platforms.

Social Clouds provide the capability to share resources among participants within a social network - leveraging on the trust relationships already existing between such participants. In such a system, users are able to trade resources between each other, rather than make use of capability offered at a (centralized) data centre. Incentives for sharing remain an important hurdle to make more effective use of such an environment, which has a significant potential for improving resource utilization and making available additional capacity that remains dormant. We utilize the socio-economic model proposed by Silvio Gesell to demonstrate how a "virtual currency" could be used to incentivise sharing of resources within a "community". We subsequently demonstrate the benefit provided to participants within such a community using a variety of economic (such as overall credits gained)and technical (number of successfully completed transactions) metrics, through simulation.

Virtualized data centers and clouds are being increasingly considered for traditional High-Performance Computing (HPC) workloads that have typically targeted Grids and conventional HPC platforms. However, maximizing energy efficiency, cost-effectiveness, and utilization of data center resources while ensuring performance and other Quality of Service (QoS) guarantees for HPC applications requires careful consideration of important and extremely challenging tradeoffs. An innovative application-centric energy-aware strategy for Virtual Machine (VM) allocation is presented. The proposed strategy ensures high resource utilization and energy efficiency through VM consolidation while satisfying application QoS. While existing VM allocation solutions are aimed at satisfying only the resource utilization requirements of applications along only one dimension (CPU utilization), the proposed approach is more generic as it employs knowledge obtained through application profiling along multiple dimensions. The results of our evaluation show that the proposed VM allocation strategy enables significant reduction either in energy consumption or in execution time, depending on the optimization goals.

Energy efficiency of large-scale data centers is becoming a major concern not only for reasons of energy conservation, failures, and cost reduction, but also because such sys tems are soon reaching the limits of power available to them. Like High Performance Computing (HPC) systems, large-scale clu ster-based data centers can consume power in megawatts, and of all the power consumed by such a system, only a fraction is used for actual computations. In this paper, we study the potential of application-centric aggressive power management of data center's resources for HPC workloads. Specifically, we consider power management mechanisms and controls (currently or soon to be) available at different levels and for different subsystems, and leverage several innovative approaches that have been taken to tackle this problem in the last few years, can be effectively used in a application-aware manner for HPC workloads. To do this, we first profile sta ndard HPC benchmarks with respect to behaviors, resource usage and power impact on individual computing nodes. Based on a power and latency model and the workload profiles, we develop an algorithm that can improve energy efficiency with little or no performance loss. We then evaluate our proposed algorithm through simulations using empirical power characterization and quantification. Finally, we validate the simulation results with actual executions on real hardware. The obtained results show that by using application aware power management, we can re-du ce the average energy consumption without significant penalty in performance. This motivates us to investigate autonomic approaches for application-aware aggressive power management and cross layer and cross function predictive subsystem level power management for large-scale data centers.

Increasing the utilization of many-core systems has been one of the forefront topics these last years. Although many-cores architectures were merely theoretical models few years ago, they have become an important part of the high performance computing market. The semiconductor industry has developed Graphical Processing Units (GPU) systems that provide access to many cores (i.e: Larrabee, Fermi or Tesla) that can be used for General Purpose (GP) computing. In this paper, we propose and evaluate a scheduling strategy for GPU and many-core architectures for HPC environments. Specifically, our strategy is a variant of the backfilling scheduling policy with resource sharing considerations. We propose a scheduling strategy that considers the differences between GP processors and GPU computing elements in terms of computational capacity and memory bandwidth. To do this, our approach uses a resource model that predicts how shared resources are used in both GP and GPU/many-core elements. Furthermore, it considers the differences between these elements in terms of performance. First, it models their differences in terms of computational power and how they share the access to the node's memory bandwidth. Second, it characterizes how the processes are allocated to the GPU. Using this resource model, we design the Power Aware resource selection policy, which we combine with the Less Consume scheduling policy. Our strategy tries to allocate jobs aiming at reducing the memory contention and the energy consumption. Results show that the scheduling strategies proposed in this work are able to save over 40\% of energy and improve the system performance up to 30\% with respect to traditional backfilling strategies.

As energy efficiency and associated costs become key concerns, consolidated and virtualized data centers and clouds are attractive computing platforms for data- and compute- intensive applications. These platforms provide an abstraction of nearly-unlimited computing resources through the elastic use of pools of consolidated resources, and provide opportunities for higher utilization and energy savings. Recently, these platforms are also being considered for more traditional high-performance computing (HPC) applications that have typically targeted Grids and similar conventional HPC platforms. However, maximizing energy efficiency, cost-effectiveness, and utilization for these applications while ensuring performance and other Quality of Service (QoS) guarantees, requires leveraging important and extremely challenging tradeoffs. These include, for example, the tradeoff between the need to efficiently create and provision Virtual Machines (VMs) on data center resources and the need to accommodate the heterogeneous resource demands and runtimes of these applications. In this paper we present an energy-aware online provisioning approach for HPC applications on consolidated and virtualized computing platforms. Energy efficiency is achieved using a workload-aware, just-right dynamic provisioning mechanism and the ability to power down subsystems of a host system that are not required by the VMs mapped to it. We evaluate the presented approach using real HPC workload traces from widely distributed production systems. The results presented demonstrated that compared to typical reactive or predefined provisioning, our approach achieves significant improvements in energy efficiency with an acceptable QoS penalty.

As energy efficiency and associated costs become key concerns, consolidated and virtualized data centers and clouds are attractive computing platforms for data- and compute-intensive applications. Recently, these platforms are also being considered for more traditional high-performance computing (HPC) applications. However, maximizing energy efficiency, cost-effectiveness, and utilization for these applications while ensuring performance and other Quality of Service (QoS) guarantees, requires leveraging important and extremely challenging tradeoffs. These include, for example, the tradeoff between the need to efficiently create and provision Virtual Machines (VMs) on data center resources and the need to accommodate the heterogeneous resource demands and runtimes of the applications that run on them. In this paper we propose an energy-aware online provisioning approach for HPC applications on consolidated and virtualized computing platforms. Energy efficiency is achieved using a workload-aware, just-right dynamic provisioning mechanism and the ability to power down subsystems of a host system that are not required by the VMs mapped to it. Our preliminary evaluations show that our approach can improve energy efficiency with an acceptable QoS penalty.

The fields of Grid, Utility and Cloud Computing have a set of common objectives in harnessing shared resources to optimally meet a great variety of demands cost-effectively and in a timely manner Since Grid Computing started its technological journey about a decade earlier than Cloud Computing, the Cloud can benefit from the technologies and experience of the Grid in building an infrastructure for distributed computing. Our comparison of Grid and Cloud starts with their basic characteristics and interaction models with clients, resource consumers and providers. Then the similarities and differences in architectural layers and key usage patterns are examined. This is followed by an in depth look at the technologies and best practices that have applicability from Grid to Cloud computing, including scheduling, service orientation, security, data management, monitoring, interoperability, simulation and autonomic support. Finally, we offer insights on how these techniques will help solve the current challenges faced by Cloud computing.

Grid meta-broker is a key enabler in realizing the full potential of inter-operating grid computing systems. A challenge to properly evaluate the effectiveness of meta-brokers is the complexity of developing a realistic grid experimental environment. In this paper, this challenge is addressed by a unique combination of two approaches: using reduced workload traces to demonstrate the resource matching and scheduling functions of the meta-broker, and using emulation to provide a flexible and scalable modeling and management for local resources of a grid environment. Real workload traces are reduced while preserving their key workload characteristics to allow exploration of various dimensions of meta-broker functions in reasonable time. Evaluation of round-robin, queue-length, and utilization based meta-broker scheduling algorithms shows that they have different effects on various workloads.

While numerous global initiatives exist to address the potential hazards posed by volcanic eruption events and assess impacts from a civil security viewpoint, there is still no single, unified, international system of early warning and hazard tracking for eruptions. Numerous gaps exist in the risk reduction cycle, from data collection, to data processing, and finally dissemination of salient information to relevant parties. As part of the International Space University's Space Studies Program during summer 2008, a detailed gap analysis of the state of volcano disaster risk reduction was undertaken. Using this analysis as a basis, a novel framework to deal with all the identified gaps is proposed as an outcome of this study. This paper presents the principal innovations concerning the data and information processing within this work. The name of the framework was chosen to be VIDA to abbreviate ‘Volcanic Activity Processing of Observation and Remote Sensing Data Integrated Data-Sharing and Analysis’. In order to present this framework, it is shown how different high performance computing resources can be orchestrated in order to support fast and efficient tools that would support VIDA in mitigating the risk posed by volcanic eruptions and other natural hazards. Furthermore, how the system could be used by describing different use cases is also presented in this paper.

Job scheduling policies for HPC centers have been extensively studied in the last few years, especially backfilling based policies. Almost all of these studies have been done using simulation tools. All the existent simulators use the runtime (either estimated or real) provided in the workload as a basis of their simulations. In our previous work we analyzed the impact on system performance of considering the resource sharing (memory bandwidth) of running jobs including a new resource model in the Alvio simulator. Based on this studies we proposed the LessConsume and LessConsume Threshold resource selection policies. Both are oriented to reduce the saturation of the shared resources thus increasing the performance of the system. The results showed how both resource allocation policies shown how the performance of the system can be improved by considering where the jobs are finally allocated. Using the LessConsume Threshold Resource Selection Policy, we propose a new backfilling strategy : the Resource Usage Aware Backfilling job scheduling policy. This is a backfilling based scheduling policy where the algorithms which decide which job has to be executed and how jobs have to be backfilled are based on a different Threshold configurations. This backfilling variant that considers how the shared resources are used by the scheduled jobs. Rather than backfilling the first job that can moved to the run queue based on the job arrival time or job size, it looks ahead to the next queued jobs, and tries to allocate jobs that would experience lower penalized runtime caused by the resource sharing saturation. In the paper we demostrate how the exchange of scheduling information between the local resource manager and the scheduler can improve substantially the performance of the system when the resource sharing is considered. We show how it can achieve a close response time performance that the shorest job first Backfilling with First Fit (oriented to improve the start time for the allocated jobs) providing a qualitative improvement in the number of killed jobs and in the percentage of penalized runtime.

Forecasting and monitoring earth phenomenon is fundamental to reducing natural disasters hazards. However, how to react after a given disaster occurs is also a key issue that can improve response times and save lives. GISCorps is a program of the Urban and Regional Information Systems Association (URISA) that coordinates short-term, volunteer-based GIS services to underprivileged communities.

Grid Resource Management tools evolved from manual discovery and task submission to sophisticated brokering solutions. User requirements created certain properties that resource managers have learned to support. This development is still continuing, and users already need to stress themselves to distinguish brokers and migrate their applications when they move to a different grid. Moreover, grid interoperability have emerged the need for higher level brokering services. This paper introduces a meta-brokering approach that means a higher level resource management by enabling automatic and simultaneous utilization of Grid Brokers. First we gather the requirements of this novel middleware service then define a general meta-brokering architecture. Finally we show how meta-brokers can be implemented in different grid environments and we conclude with their evaluations.

Grid computing has emerged as a way to share geographically and organizationally distributed resources that may belong to different institutions or administrative domains. In this context, the scheduling and resource management is usually performed by a grid resource broker. The scheduling task consists of distributing the jobs among the different centers resources and the need to coordinate the grid with the underlying scheduling levels which have already been identified. However, there is still a lack of policies for this approach. In this paper we describe and evaluate our coordinated grid scheduling strategy. We take as a reference the FCFS job scheduling policy and the matchmaking approach for the resource selection. We also present a new job scheduling policy based on backfilling (JR-backfilling) that aims to improve the workloads execution performance, avoiding starvation and the SLOW-coordinated resource selection policy that considers the average bounded slowdown of the resources as the main parameter to perform the resource selection. From our evaluation, based on trace-driven simulations of real grid systems, we state that our proposed coordinated strategy can substantially improve the workloads execution performance as well as the resource utilization.

Grid resource management tools have evolved from manual discovery and job submission to sophisticated brokering solutions. User requirements have created certain properties that resource managers have learned to support. This development is still continuing, and users already find it difficult to distinguish brokers and to migrate their applications when they move to a different grid. Moreover, new architectures are continuously being proposed, such as multi-site and interoperable grid systems. This paper presents the Alvio simulation framework which is designed to evaluate job scheduling strategies in complex HPC infrastructures. The main contribution of this simulator is that it allows modeling from local systems to interoperable grid scenarios. We also present an evaluation of multi-site and grid interoperable systems which shows the effect of job forwarding between different brokers.

Grid computing supports workload execution on computing resources that are shared across a set of collaborative organizations. At the core of workload management for grid computing is a software component, called meta-scheduler or grid resource broker, that provides a virtual layer on top of heterogeneous grid middleware, schedulers, and resources. Meta-schedulers typically enable end-users and applications to compete over distributed shared resources through the use of one or more instances of the same meta-scheduler, in a centralized or distributed manner, respectively. We propose an approach to enabling autonomic meta-scheduling through the use of a new communication protocol that -if adopted by different meta-schedulers or by the applications using them- can improve the workload execution while avoiding potential chaos, which can be resulted from blind competition over resources. This can be made possible by allowing the meta- schedulers and/or their applications to engage in a process to negotiate their roles (e.g., consumer, provider, or both), scheduling policies, service-level agreement, etc. To show the feasibility of our approach, we developed a prototype that enables some preliminary autonomic management among three different meta-schedulers, namely, GridWay, eNANOS, andTDWB.

Grid computing supports shared access to computing resources from cooperating organizations or institutes in the form of virtual organizations. Resource brokering middleware, commonly known as a meta-scheduler or a resource broker, matches jobs to distributed resources. Recent advances in meta- scheduling capabilities are extended to enable resource matching across multiple virtual organizations. Several architectures have been proposed for interoperating meta-scheduling systems. This paper presents a hybrid approach, combining hierarchical and peer-to-peer architectures for flexibility and extensibility of these systems. A set of protocols are introduced to allow different meta-scheduler instances to communicate over Web Services. Interoperability between three heterogeneous and distributed organizations (namely, BSC, FIU, and IBM), each using different meta-scheduling technologies, is demonstrated under these protocols and resource models.

In large Grids, like the National Grid Service (NGS), or large distributed architecture different scheduling entities are involved. Despite a global scheduling approach would archive higher performance and could increment the utilization of global system in these scenarios usually independent schedulers carry out its own scheduling decisions. In this paper we present how a coordinated scheduling among all the different centers using data mining prediction techniques can substantially improve the performance of the global distributed infrastructure, and can provide a uniform access to the user to all the heterogeneous Grid resources. We present the Grid Backfilling meta-scheduling policy that optimizes the global utilization of the system resources and increases substantially the response time for the jobs. We also present how data mining techniques applied to historical information can provide very suitable inputs for carrying out the Grid Backfilling meta-scheduling decisions.

Grids allow large scale resource-sharing across different administrative domains. Those diverse resources are likely to join or quit the Grid at any moment or possibly to break down. Grid monitoring tools have to adapt supporting access information to these heterogeneous and not reliable environments. There is a wide rage of types of resources to be monitored or entities, with different nature, characteristics and so on. These issues make the task of gathering Grid information complex to treat, and it is difficult to provide a generals ways for accessing to all this information. In this paper we propose a set of functionalities that a Grid Information System should provide. We describe the Palantir meta-information system that has been designed for uniform the access to different monitoring and information systems and that implements all the discussed functionalities. Moreover, we present real examples that state how Palantir has been integrated providing the uniform. access to systems with heterogeneous information providers.

Since the management and the optimal utilization of the highly dynamic grid resources cannot be handled by the users themselves, various grid Resource Brokers have been developed, supporting different grids. To ease interoperability and the higher level utilization of different resource brokers, we introduce a metadata model for storing broker capabilities and show how an implementation of this model can be realized. We believe that this abstraction will help standardizing inter-broker communication to enable more efficient grid resource utilization.

The impact of hurricanes is so devastating throughout different levels of society that there is a pressing need to provide a range of users with accurate and timely information that can enable effective planning for and response to potential hurricane landfalls. The Weather Research and Forecasting (WRF) code is the latest numerical model that has been adopted by meteorological services worldwide. The current version of WRF has not been designed to scale out of a single organization's local computing resources. However, the high resource requirements of WRF for fine-resolution and ensemble forecasting demand a large number of computing nodes, which typically cannot be found within one organization. Therefore, there is a pressing need for the Grid-enablement of the WRF code such that it can utilize resources available in partner organizations. In this paper, we present our research on Grid enablement of WRF by leveraging our work in transparent shaping, GRID superscalar, profiling, code inspection, code modeling, meta-scheduling, and job flow management.