Search

With increasing availability and use of Internet of Things (IoT) devices large amounts of streaming data is now being produced at high velocity. Applications which require low latency response such as video surveillance demand a swift and efficient analysis of this data. Existing approaches employ cloud infrastructure to store and perform machine learning based analytics on this data. This centralized approach has limited ability to support analysis of real-time, large-scale streaming data due to network bandwidth and latency constraints between data source and cloud. We propose RealEdgeStream (RES) an edge enhanced stream analytics system for large-scale, high performance data analytics. The proposed approach investigates the problem of video stream analytics by proposing (i) filtration and (ii) identification phases. The filtration phase reduces the amount of data by filtering low value stream objects using configurable rules. The identification phase uses deep learning inference to perform analytics on the streams of interest. The stages are mapped onto available in-transit and cloud resources using a placement algorithm to satisfy the Quality of Service (QoS) constraints identified by a user. The job completion in the proposed system takes 49\% less time and saves 99\% bandwidth compared to a centralized cloud-only based approach.

The landscape for cloud services and cyberinfrastructure offerings has increased drastically over the past few years. Initially, users moved their applications to the cloud to take advantage of a pay-per-usage model and on-demand access. However, as more cloud providers joined the market, users shifted their goals for using cloud computing from cost reduction to resilience, agility, and optimization. These goals can be achieved by dynamically combining services from multiple providers, for example, to avoid data center or cloud zone outages or to take advantage of extensive offerings with different price points. However, to efficiently support application deployment in this dynamic environment, new models and tools that can measure the application performance and the Quality of Service (QoS) of different configurations are required. The goal of this work is to evaluate the application performance and the QoS of a distributed Software-Defined Environment as well as calculate the QoS of alternative configurations from the underlying pool of services. In particular, we present a mathematical model and a tool for evaluating the performance and QoS of batch application workflows in a distributed environment. We experimentally evaluate the proposed model using a bioinformatics workflow running on infrastructure services from multiple cloud providers.

With the increasing availability of edge computing resources, there is a need to develop edge orchestration and resource management techniques to support application resilience and performance. Similar to the use of containers and microservices for cloud environments, it is important to understand the key attributes that characterize edge native applications. As edge devices increase in their autonomy and intelligence, orchestration techniques are needed to respond to changes in device properties, availability, security credentials, migration, and network connectivity protocols. Implementing autonomics techniques for edge computing can increase the resilience of the interaction between devices and applications reducing execution time and cost. The use of autonomics at the network edge can address the complexity requirement of industrial workflows to overcome execution latency, data privacy, and reliability constraints.

Large-scale scientific facilities provide a broad community of researchers and educators with open access to instrumentation and data products generated from geographically distributed instruments and sensors. This paper discusses key architectural design, deployment, and operational aspects of a production cyberinfrastructure for the acquisition, processing, and delivery of data from large scientific facilities using experiences from the National Science Foundation's Ocean Observatories Initiative. This paper also outlines new models for data delivery and opportunities for insights in a wide range of scientific and engineering domains as the volumes and variety of data from facilities grow.

The Virtual Data Collaboratory is a federated data cyberinfrastructure designed to drive data-intensive, interdisciplinary, and collaborative research that will impact researchers, educators, and entrepreneurs across a broad range of disciplines and domains as well as institutional and geographic boundaries.

The Internet of Things fosters an emerging class of analytics that connects sensors, vehicles, industries, and consumers through the internet to enable scientific and industrial applications. These services require a large computing capacity to perform well, while often being under the constraints to move data from the edge of the network to the cloud. Also, they require system support to program reactions that occur at runtime, especially when the target infrastructure capacities and capabilities are unknown during the design. The core of this survey is a comprehensive review of existing components for Edge Computing and Internet of Things applications. In recent years, the landscape of the edge middleware platforms has grown exponentially with more than a hundred available solutions in academia and industry. Such platforms are required to provide necessary components for sensor registration, resource discovery, workflow composition, and data processing. In this regard, we surveyed existing solutions through the lens of a simplified three-layer architecture for the design of edge-based middleware, along with design goals for each of the proposed layers. The paper concludes with some open challenges and possible future research directions.

The articles in this special section focus on advancements of utility and cloud computing technologies and services. Computing is rapidly moving towards a model where it is provided as services that are delivered in a manner similar to traditional utilities such as water, electricity, gas, and telephony. In such a model, users access services according to their requirements, without regard to where the services are hosted or how they are delivered. Several computing architectures have evolved to realize this utility computing vision, including Grid computing, Service- Oriented Architecture (SOA) and Cloud computing,which has recently shifted into the center of attention in the ICT industry. Increasing numbers of IT vendors are promising to offer applications, storage and computation hosting services with conforming Service-Level Agreements

Dramatic changes in the technology landscape marked by increasing scales and pervasiveness of compute and data have resulted in the proliferation of edge applications aimed at effectively processing data in a timely manner. As the levels and fidelity of instrumentation increases and the types and volumes of available data grow, new classes of applications are being explored that seamlessly combine real-time data with complex models and data analytics to monitor and manage systems of interest. However, these applications require a fluid integration of resources at the edge, the core, and along the data path to support dynamic and data-driven application workflows, that is, they need to leverage a computing continuum. In this article, we present our vision for enabling such a computing continuum and specifically focus on enabling edge-to-cloud integration to support data-driven workflows. The research is driven by an online data-driven tsunami warning use case that is supported by the deployment of large-scale national environment observation systems. This article presents our overall approach as well as current status and next steps.

Combining edge processing (at data capture site) with analysis carried out while data is enroute from the capture site to a data center offers a variety of different processing models. Such in-transit nodes include network data centers that have generally been used to support content distribution (providing support for data multicast and caching), but have recently started to offer user-defined programmability, through Software Defined Networks (SDN) capability, e.g., OpenFlow and Network Function Visualization (NFV). We demonstrate how this multi-site computational capability can be aggregated to support video analytics, with Quality of Service and cost constraints (e.g., latency-bound analysis). The use of SDN technology enables separation of the data path from the control path, enabling in-network processing capabilities to be supported as data is migrated across the network. We propose to leverage SDN capability to gain control over the data transport service with the purpose of dynamically establishing data routes such that we can opportunistically exploit the latent computational capabilities located along the network path. Using a number of scenarios, we demonstrate the benefits and limitations of this approach for video analysis, comparing this with the baseline scenario of undertaking all such analysis at a data center located at the core of the infrastructure.

International audience; Internet of Things (IoT) is bringing an increasing number of connected devices that have a direct impact on the growth of data and energy-hungry services. These services are relying on Cloud infrastructures for storage and computing capabilities, transforming their architecture into more a distributed one based on edge facilities provided by Internet Service Providers (ISP). Yet, between the IoT device, communication network and Cloud infrastructure, it is unclear which part is the largest in terms of energy consumption. In this paper, we provide end-to-end energy models for Edge Cloud-based IoT platforms. These models are applied to a concrete scenario: data stream analysis produced by cameras embedded on vehicles. The validation combines measurements on real test-beds running the targeted application and simulations on well-known sim-ulators for studying the scaling-up with an increasing number of IoT devices. Our results show that, for our scenario, the edge Cloud part embedding the computing resources consumes 3 times more than the IoT part comprising the IoT devices and the wireless access point.

During disaster response, large volumes and diverse types of data sets are often continuously generated, and in many case these data sets create overwhelming burdens to data processing infrastructure and teams. At the same time, decision making during disaster response requires timely and relevant information which has to be extracted as expeditiously as possible from these large data sets. Therefore, processing of disaster related data sets is often time sensitive and requires coordination and prioritization. To accomplish this, we propose a two-stage approach to facilitate efficient and effective data processing for disaster decision support. In the first stage, a Data Envelope Analysis (DEA) model is introduced to model the articulation process about information needs such that providing a formal way of prioritizing data processing task. In the second stage, the prioritized data processing workflow is implemented on an Apache Storm based streaming processing platform in the EC2 cloud, with a focus on computational resource optimization. To validate the proposed approach, a Hurricane Sandy based use case was used to evaluate the performance of the proposed approach. Results show that our approach can compute up to 69% (three supervisor nodes) faster than a conventional serial processing approach.

Cloud computing is emerging as a viable platform for scientific exploration. Elastic and on-demand access to resources (and other services), the abstraction of “unlimited” resources, and attractive pricing models provide incentives for scientists to move their workflows into clouds. Generalizing these concepts beyond a single virtualized datacenter, it is possible to create federated marketplaces where different types of resources (e.g., clouds, HPC grids, supercomputers) that may be geographically distributed, are collectively exposed as a single elastic infrastructure. This presents opportunities for optimizing the execution of application workflows with heterogeneous and dynamic requirements, and tackling larger scale problems. In this paper, we introduce a framework to manage the end-to-end execution of data-intensive application workflows in dynamic software-defined resource federation. This framework enables the autonomic execution of workflows by elastically provisioning an appropriate set of resources that meet application requirements, and by adapting this set of resources at runtime as the requirements change. It also allows users to customize scheduling policies that drive the way resources federated and used. To demonstrate the benefits of our approach, we study the execution of two different data-intensive scientific workflows in a multi-cloud federation using different policies and objective functions.

As the number of people who interact on social networks increases, and coupled with the greater capability made available within our computational devices, there is the potential to establish “Social Clouds”—a resource sharing infrastructure that enable people who have trust relationships to come together to share computational/ data services within a community. Social clouds can also provide the means to enhance multi-user collaboration and greatly stimulate the exchange of resources among participants. Recent research in the establishment and use of Social Clouds has raised significant interest by proposing an environment where users are able to trade resources mediated by a social networking mechanism. In such a cloud environment the incentives for sharing can represent a solution for improving resource utilisation and for making available additional capacity to friends and collaborators. In this paper we demonstrate how revenue can be earned within a social cloud community, by executing internal (intra community) and external (inter community) tasks. A number of different scenarios are first investigated through simulation, using the PeerSim simulator, in order to validate our approach. We use two key metrics: revenue and reputation, to evaluate how the system dynamics change as new tasks are added to one or more communities for execution, along with additional behaviours, such as nodes migrating from one community to another, or selectively reporting on the outcome of task execution. Subsequently, we develop a practical deployment using a federated cloud scenario using the CometCloud system—deployed over three sites: Cardiff (UK), Rutgers and Indiana. We show how approaches that have been simulated in PeerSim can be implemented in practice.

International audience; The number of Internet of Things applications is forecast to grow exponentially within the coming decade. Owners of such applications strive to make predictions from large streams of complex input in near real time. Cloud-based architectures often centralize storage and processing, generating high data movement overheads that penalize real-time applications. Edge and Cloud architecture pushes computation closer to where the data is generated, reducing the cost of data movements and improving the application response time. The heterogeneity among the edge devices and cloud servers introduces an important challenge for deciding how to split and orchestrate the IoT applications across the edge and the cloud. In this paper, we extend our IoT Edge Framework, called R-Pulsar, to propose a solution on how to split IoT applications dynamically across the edge and the cloud, allowing us to improve performance metrics such as end-to-end latency (response time), bandwidth consumption, and edge-to-cloud and cloud-to-edge messaging cost. Our approach consists of a programming model and real-world implementation of an IoT application. The results show that our approach can minimize the end-to-end latency by at least 38% by pushing part of the IoT application to the edge. Meanwhile, the edge-to-cloud data transfers are reduced by at least 38% and the messaging costs are reduced by at least 50% when using the existing commercial edge cloud cost models.

The performance of modern Big Data frameworks, e.g. Spark, depends greatly on high-speed storage and shuffling, which impose a significant memory burden on production data centers. In many production situations, the persistence and shuffling intensive applications can suffer a major performance loss due to lack of memory. Thus, the common practice is usually to over-allocate the memory assigned to the data workers for production applications, which in turn reduces overall resource utilization. One efficient way to address the dilemma between the performance and cost efficiency of Big Data applications is through data center computing resource disaggregation. This paper proposes and implements a system that incorporates the Spark Big Data framework with a novel in-memory distributed file system to achieve memory disaggregation for data persistence and shuffling. We address the challenge of optimizing performance at affordable cost by co-designing the proposed in-memory distributed file system with large-volume DIMM-based persistent memory (PMEM) and RDMA technology. The disaggregation design allows each part of the system to be scaled independently, which is particularly suitable for cloud deployments. The proposed system is evaluated in a production-level cluster using real enterprise-level Spark production applications. The results of an empirical evaluation show that the system can achieve up to a 3.5- fold performance improvement for shuffle-intensive applications with the same amount of memory, compared to the default Spark setup. Moreover, by leveraging PMEM, we demonstrate that our system can effectively increase the memory capacity of the computing cluster with affordable cost, with a reasonable execution time overhead with respect to using local DRAM only.

Estimating energy costs for an industrial process can be computationally intensive and time consuming, especially as it can involve data collection from different (distributed) monitoring sensors. Industrial processes have an implicit complexity involving the use of multiple appliances (devices/ sub-systems) attached to operation schedules, electrical capacity and optimisation setpoints which need to be determined for achieving operational cost objectives. Addressing the complexity associated with an industrial workflow (i.e. range and type of tasks) leads to increased requirements on the computing infrastructure. Such requirements can include achieving execution performance targets per processing unit within a particular size of infrastructure i.e. processing & data storage nodes to complete a computational analysis task within a specific deadline. The use of ensemblebased edge processing is identifed to meet these Quality of Service targets, whereby edge nodes can be used to distribute the computational load across a distributed infrastructure. Rather than relying on a single edge node, we propose the combined use of an ensemble of such nodes to overcome processing, data privacy/ security and reliability constraints. We propose an ensemble-based network processing model to facilitate distributed execution of energy simulations tasks within an industrial process. A scenario based on energy profiling within a fisheries plant is used to illustrate the use of an edge ensemble. The suggested approach is however general in scope and can be used in other similar application domains.

Applying deep learning models to large-scale IoT data is a compute-intensive task and needs significant computational resources. Existing approaches transfer this big data from IoT devices to a central cloud where inference is performed using a machine learning model. However, the network connecting the data capture source and the cloud platform can become a bottleneck. We address this problem by distributing the deep learning pipeline across edge and cloudlet/fog resources. The basic processing stages and trained models are distributed towards the edge of the network and on in-transit and cloud resources. The proposed approach performs initial processing of the data close to the data source at edge and fog nodes, resulting in significant reduction in the data that is transferred and stored in the cloud. Results on an object recognition scenario show 71\% efficiency gain in the throughput of the system by employing a combination of edge, in-transit and cloud resources when compared to a cloud-only approach.

Reproducibility is a foundation of solid scientific and technical research. The ability to repeat research is key to confirming the validity of a scientific discovery. At IEEE Transactions on Parallel and Distributed Systems (TPDS), we are committed to enabling reproducible research through transparency and the availability and potential reuse of code. We have partnered with Code Ocean, New York, a cloud-based computational reproducibility platform, to pilot the post-publication peer review of code associated with papers published in TPDS. Authors who have published in TPDS can make their work more reproducible and earn a reproducibility badge by submitting their associated code for post-publication peer review.

Multi-cloud systems have enabled resource and service providers to co-exist in a market where the relationship between clients and services depends on the nature of an application and can be subject to a variety of different Quality of Service (QoS) constraints. Deciding whether a cloud provider should host (or finds it profitable to host) a service in the long-term would be influenced by parameters such as the service price, the QoS guarantees required by customers, the deployment cost (taking into account both cost of resource provisioning and operational expenditure, e.g. energy costs) and the constraints over which these guarantees should be met. In a federated cloud system users can combine specialist capabilities offered by a limited number of providers, at particular cost bands—such as availability of specialist co-processors and software libraries. In addition, federation also enables applications to be scaled on-demand and restricts lock in to the capabilities of a particular provider. We devise a market model to support federated clouds and investigate its efficiency in two real application scenarios:(i) energy optimisation in built environments and (ii) cancer image processing both requiring significant computational resources to execute simulations. We describe and evaluate the establishment of such an application based federation and identify a cost-decision based mechanism to determine when tasks should be outsourced to external sites in the federation. The following contributions are provided: (i) understanding the criteria for accessing sites within a federated cloud dynamically, taking into account factors such as performance, cost, user perceived value, and specific application requirements; (ii) developing and deploying a cost based federated cloud framework for supporting real applications over three federated sites at Cardiff (UK), Rutgers and Indiana (USA), (iii) a performance analysis of the application scenarios to determine how task submission could be supported across these three sites, subject to particular revenue targets.

The AEC (Architecture Engineering and Construction) sector is a highly fragmented, data intensive, project based industry, involving a number or very different professions and organisations. The industry's strong data sharing and processing requirements means that the management of building data is complex and challenging. We present a data sharing capability utilising Cloud Computing, with two key contributions: 1) a governance model for building data, based on extensive research and industry consultation. This governance model describes how individual data artefacts within a building information model relate to each other and how access to this data is controlled; 2) a prototype implementation of this governance model, utilising the CometCloud autonomic cloud computing engine, using the Master/Work paradigm. This prototype is able to successfully store and manage building data, provide security based on a defined policy language and demonstrate scale-out in case of increasing demand or node failure. Our prototype is evaluated both qualitatively and quantitatively. To enable this evaluation we have integrated our prototype with the 3D modelling software—Google Sketchup. We also evaluate the prototype's performance when scaling to utilise additional nodes in the Cloud and to determine its performance in case of node failures.

The Internet of Things (IoT) describes the emerging paradigm that connects sensors, often located at the edge of the network, to stream processing engines located at the core of the network to enable online data-driven monitoring, management, and control. As IoT applications require increasing volumes of streaming data to be processed by complex workflows in a timely manner, it is becoming important to also leverage resources closer to the edge. Furthermore, the topology of these workflows and where theyare executed is determined not only by application objectives and available resources, but also by the content of the data streams, however, current stream processing engines do not provide this flexibility. In this paper, we present a programming framework that enables applications to specify data-driven, location- and resource-aware processing of data streams. Specifically, it provides abstractions for specifying where and how a data stream is processed based on its content, spatial and temporal characteristics. We also present an implementation of the framework using an event-driven runtime, where events are associatively described. Finally, we demonstrate the effectiveness of the solution by an evaluation of scalability and performance using a disaster response application usecase.

International audience; The emergence of Internet of Things (IoT) is participating to the increase of data- and energy-hungry applications. As connected devices do not yet offer enough capabilities for sustaining these applications, users perform computation offloading to the cloud. To avoid network bottlenecks and reduce the costs associated to data movement, edge cloud solutions have started being deployed, thus improving the Quality of Service. In this paper, we advocate for leveraging on-site renewable energy production in the different edge cloud nodes to green IoT systems while offering improved QoS compared to core cloud solutions. We propose an analytic model to decide whether to offload computation from the objects to the edge or to the core Cloud, depending on the renewable energy availability and the desired application QoS. This model is validated on our application use-case that deals with video stream analysis from vehicle cameras.

The popularity and proliferation of the Internet of Things (IoT) paradigm is resulting in a growing number of devices connected to the Internet. These devices are generating and consuming unprecedented amounts of data at the edges of the infrastructure, and are enabling new classes of data-drivenapplications, however, current approaches typically rely on cloud platforms located at the core of the infrastructure to process data. As the number of devices and the amount of data they generate and consume increases, such core-centric approaches are becoming increasingly inefficient as they need to transfer data back and forth between the edge and the core. Furthermore, the latencies associated with such data transfer may not be able to support applications involving time-critical data-driven decision making. In this paper, we propose an edge-based programming framework that allows users to define how data streams are processed based on the content and the location of the data. This enables the definition of data-driven reactive behaviors that can effectively exploit data patterns to dynamically drive stream processing, leveraging resources located at the edges of the infrastructure. We have implemented a prototype of the proposed approach and performed several experiments to evaluate its scalability and efficiency against a more typical single-cloud approach. Using a smart-city application usecase, we illustrate that the presented programming system can support data-driven stream processing using edge resources. In terms of scalability, our experiments show that the system can scale to hundreds of nodes while keeping overheads low. Our experiments also show that our approach can perform up to 56% better than a single cloud approach that does not consider data and user locality.

Service-based access models coupled with emerging application deployment technologies are enabling opportunities for realizing highly customized software-defined environments, which can support dynamic and data-driven applications. However, this requires rethinking traditional resource federation models to support dynamic resource compositions, which can adapt to evolving application needs and the dynamic state of underlying resources. In this paper, we present a programmable approach that leverages software-defined techniques to create a dynamic space-time infrastructure service composition. We propose the use of Constraint Programming as a formal language to allow users, applications, and service providers to define the desired state of the execution environment. The resulting distributed software-defined environment continually adapts to meet objectives/constraints set by the users, applications, and/or resource providers. We present the design and prototype implementation of such distributed software-defined environment. We use a cancer informatics workflow to demonstrate the operation of our framework using resources from five different cloud providers, which are aggregated on-demand based on dynamic user and resource provider constraints.

Recent advances in the type and variety of sensing technologies have led to an extraordinary growth in the volume of data being produced and led to a number of streaming applications that make use of this data. Sensors typically monitor environmental or physical phenomenon at predefined time intervals or triggered by user-defined events. Understanding how such streaming content (the raw data or events) can be processed within a time threshold remains an important research challenge. We investigate how a cloud-based computational infrastructure can autonomically respond to such streaming content, offering quality of service guarantees. In particular, we contextualize our approach using an electric vehicles (EVs) charging scenario, where such vehicles need to connect to the electrical grid to charge their batteries. There has been an emerging interest in EV aggregators (primarily intermediate brokers able to estimate aggregate charging demand for a collection of EVs) to coordinate the charging process. We consider predicting EV charging demand as a potential workload with execution time constraints. We assume that an EV aggregator manages a number of geographic areas and a pool of computational resources of a cloud computing cluster to support scheduling of EV charging. The objective is to ensure that there is enough computational capacity to satisfy the requirements for managing EV battery charging requests within specific time constraints.

Recent advances in sensor technologies and instrumentation have led to an extraordinary growth of data sources and streaming applications. A wide variety of devices, from smart phones to dedicated sensors, have the capability of collecting and streaming large amounts of data at unprecedented rates. A number of distinct streaming data models have been proposed. Typical applications for this include smart cites & built environments for instance, where sensor-based infrastructures continue to increase in scale and variety. Understanding how such streaming content can be processed within some time threshold remains a non-trivial and important research topic. We investigate how a cloud-based computational infrastructure can autonomically respond to such streaming content, offering Quality of Service guarantees. We propose an autonomic controller (based on feedback control and queueing theory) to elastically provision virtual machines to meet performance targets associated with a particular data stream. Evaluation is carried out using a federated Cloud-based infrastructure (implemented using CometCloud)-where the allocation of new resources can be based on: (i) differences between sites, i.e., types of resources supported (e.g., GPU versus CPU only), (ii) cost of execution; (iii) failure rate and likely resilience, etc. In particular, we demonstrate how Little's Law-a widely used result in queuing theory-can be adapted to support dynamic control in the context of such resource provisioning.

Fog computing provides a distributed infrastructure at the edges of the network, resulting in low-latency access and faster response to application requests when compared to centralized clouds. With this new level of computing capacity introduced between users and the data center-based clouds, new forms of resource allocation and management can be developed to take advantage of the Fog infrastructure. A wide range of applications with different requirements run on end-user devices, and with the popularity of cloud computing many of them rely on remote processing or storage. As clouds are primarily delivered through centralized data centers, such remote processing/storage usually takes place at a single location that hosts user applications and data. The distributed capacity provided by Fog computing allows execution and storage to be performed at different locations. The combination of distributed capacity, the range and types of user applications, and the mobility of smart devices require resource management and scheduling strategies that takes into account these factors altogether. We analyze the scheduling problem in Fog computing, focusing on how user mobility can influence application performance and how three different scheduling policies, namely concurrent, FCFS, and delay-priority, can be used to improve execution based on application characteristics.

The complexity of many problems in science and engineering requires computational capacity exceeding what the average user can expect from a single computational center. While many of these problems can be viewed as a set of independent tasks, their collective complexity easily requires millions of core-hours on any high-power computing (HPC) resource, and throughput that can't be sustained by a single, multiuser queuing system. An exploration of the use of aggregated HPC resources to solve large-scale engineering problems shows that it's possible to build a computational federation that's easy for end users to implement, and is elastic, resilient, and scalable. Here, the authors argue that the fusion of federated computing and real-life engineering problems can be brought to the average user if relevant middleware is provided. They report on the use of federation of 10 distributed heterogeneous HPC resources to perform a large-scale interrogation of the parameter space in the microscale fluid flow problem.

The advantages of on-demand resource availability are making cloud computing a viable platform option for research and education that may enable new practices in science and engineering.

Recent advances in sensor technologies and instrumentation have led to an extraordinary growth of data sources and streaming applications. A wide variety of devices, from smart phones to dedicated sensors, have the capability of collecting and streaming data at unprecedented rates. Typical applications include smart cities & built environments for instance, where sensor-based infrastructures continue to increase in scale and variety. Analysis of stream data involves: (i) execution of a number of operations on a time/sample window – e.g. min./max./avg., filtering, etc, (ii) a need to combine a number of such operations together, (iii) event-driven execution of operations, generally over short time durations, (iv) operation correlations across multiple data streams. The use of such operations does not fit well in the per-hour or per-minute cloud billing models currently available from cloud providers – with some notable exceptions (e.g. Amazon AWS). In this paper we discuss how micro-billing and sub-second resource allocation can be used in the context of streaming applications and how micro-billing models bring challenges to capacity management on cloud infrastructures.

Clouds are rapidly joining high-performance computing (HPC) systems, clusters, and grids as viable platforms for scientific exploration and discovery. As a result, understanding application formulations and usage modes that are meaningful in such a hybrid infrastructure, and how application workflows can effectively utilize it, is critical. Here, three hybrid HPC/grid and cloud cyber infrastructure usage modes are explored: HPC in the Cloud, HPC plus Cloud, and HPC as a Service, presenting illustrative scenarios in each case and outlining benefits, limitations, and research challenges.

Social Clouds provide the capability to share resources among participants within a social network-leveraging on the trust relationships already existing between such participants. In such a system, users are able to trade resources between each other rather than make use of capability offered at a centralized data center. Although such an environment has significant potential for improving resource utilization and making available additional capacity that remains dormant, incentives for sharing remain an important hurdle limiting its effective. In this paper, we utilize the socioeconomic model proposed by Silvio Gesell to demonstrate how a 'virtual currency' can be used to incentivise sharing of resources within a 'community'. We subsequently demonstrate, through simulations, the benefit provided to participants within such a community using a variety of economic such as overall credits gained and technical number of successfully completed transactions metrics. Further, we describe our implementation of such a Social Cloud using CometCloud. CometCloud is an autonomic computing engine for cloud and grid environments. It supports highly heterogeneous and dynamic federated cloud/Grid infrastructures, integration of public/private clouds and autonomic cloudbursts. We demonstrate the implementation of two designs on the basis of the master/worker approach: i one tuple space per cluster and ii one coordination tuple space and multiple transient spaces-one per each cluster. Finally, we discuss an extended version of our Social Cloud model where intermediary relay nodes take on more active roles as traders in a transaction. Copyright © 2013 John Wiley & Sons, Ltd.

Software-defined networks (SDNs) have proven to be an efficacious tool for undertaking complex data analysis and manipulation within data intensive applications. SDN technology allows us to separate the data path from the control path, enabling in-network processing capabilities to be supported as data is migrated across the network. We propose to leverage software defined networking to gain control over the data transport service with the purpose of dynamically establishing data routes such that we can opportunistically exploit the latent computational capabilities located along the network path. This strategy allows us to minimize waiting times at the destination data center and to cope with spikes in demand for computational capability. We validate our approach using a smart building application in a multi cloud infrastructure. We show how the in-transit processing strategy increases the computational capabilities of the infrastructure and influences the percentage of job completion without significantly impacting costs and overheads.

Cloud computing has emerged as attractive platform for computing data intensive applications. However, efficient computation of this kind of workloads requires understanding how to store, process, and analyse large volumes of data in a timely manner. Many “smart cities” applications, for instance, identify how data from building sensors can be combined together to support applications such as emergency response, energy management, etc. Enabling sensor data to be transmitted to a cloud environment for processing provides a number of benefits, such as scalability and on-demand provisioning of computational resources. In this chapter, we propose the use of a multi-layer cloud infrastructure that distributes processing over sensing nodes, multiple intermediate/gateways nodes, and large data centres. Our solution aims at utilising the pervasive computational capabilities located at the edge of the infrastructure and along the data path to reduce data movement to large data centres located “deep” into the infrastructure and perform a more efficient use of computing and network resources.

We introduce an architecture for undertaking data processing across multiple layers of a distributed computing infrastructure, composed of edge devices (making use of Internet-of-Things (IoT) based protocols), intermediate gateway nodes and large scale data centres. In this way, data processing that is intended to be carried out in the data centre can be pushed to the edges of the network -- enabling more efficient use of data centre and in-network resources. We suggest the need for specialist data analysis and management algorithms that are resource-aware, and are able to split computation across these different layers. We propose a coordination mechanism that is able to combine different types of data processing capability, such as in-transit and in-situ. An application scenario is used to illustrate the concepts, subsequently evaluated through a multi-site deployment.

Virtualized datacenters and clouds are being increasingly considered for traditional High-Performance Computing (HPC) workloads that have typically targeted Grids and conventional HPC platforms. However, maximizing energy efficiency and utilization of datacenter resources, and minimizing undesired thermal behavior while ensuring application performance and other Quality of Service (QoS) guarantees for HPC applications requires careful consideration of important and extremely challenging tradeoffs. Virtual Machine (VM) migration is one of the most common techniques used to alleviate thermal anomalies (i.e., hotspots) in cloud datacenter servers as it reduces load and, hence, the server utilization. In this article, the benefits of using other techniques such as voltage scaling and pinning (traditionally used for reducing energy consumption) for thermal management over VM migrations are studied in detail. As no single technique is the most efficient to meet temperature/performance optimization goals in all situations, an autonomic approach that performs energy-efficient thermal management while ensuring the QoS delivered to the users is proposed. To address the problem of VM allocation that arises during VM migrations, an innovative application-centric energy-aware strategy for Virtual Machine (VM) allocation is proposed. The proposed strategy ensures high resource utilization and energy efficiency through VM consolidation while satisfying application QoS by exploiting knowledge obtained through application profiling along multiple dimensions (CPU, memory, and network bandwidth utilization). To support our arguments, we present the results obtained from an experimental evaluation on real hardware using HPC workloads under different scenarios.

Emerging lightweight cloud technologies, such as Docker containers, are gaining wide traction in IT due to the fact that they allow users to deploy applications in any environment faster and more efficiently than using virtual machines. However, current Docker-based container deployment solutions are aimed at managing containers in a single-site, which limits their capabilities. As more users look to adopt Docker containers in dynamic, heterogenous environments, the ability to deploy and effectively manage containers across multiple clouds and data centers becomes of utmost importance. In this paper, we propose a prototype framework, called C-Ports, that enables the deployment and management of Docker containers across multiple hybrid clouds and traditional clusters while taking into consideration user and resource provider objectives and constraints. The framework leverages a constraint-programming model for resource selection and uses CometCloud to allocate/deallocate resources as well as to deploy containers on top of these resources. Our prototype has been effectively used to deploy and manage containers in a dynamic federation composed of five clouds and two clusters.

In this paper, we present a programming and runtime framework that enables the autonomic management of complex application workflows on hybrid computing infrastructures. The framework is designed to address system and application heterogeneity and dynamics to ensure that application objectives and constraints are satisfied. The need for such autonomic system and application management is becoming critical as computing infrastructures become increasingly heterogeneous, integrating different classes of resources from high-end HPC systems to commodity clusters and clouds. For example, the framework presented in this paper can be used to provision the appropriate mix of resources based on application requirements and constraints. The framework also monitors the system/application state and adapts the application and/or resources to respond to changing requirements or environment. To demonstrate the operation of the framework and to evaluate its ability, we employ a workflow used to characterize an oil reservoir executing on a hybrid infrastructure composed of TeraGrid nodes and Amazon EC2 instances of various types. Specifically, we show how different applications objectives such as acceleration, conservation and resilience can be effectively achieved while satisfying deadline and budget constraints, using an appropriate mix of dynamically provisioned resources. Our evaluations also demonstrate that public clouds can be used to complement and reinforce the scheduling and usage of traditional high performance computing infrastructure.

Since the social media hype started in the early 2000s, the Internet has bloomed with user-generated data. The content generated by users in social media varies from blogs, forums, social network platforms, and video sharing communities. This data has a special emphasis on the relationships among users of the community. As a consequence, social media data contains significant information about their creators and people around them. For this reason, law enforcement and insurance companies, among others, are starting to explore ways of using this data to identify unlawful or fraudulent activities. In this work, we present a solution to extract and analyze social media data in pursuit of identifying insurance fraud. We describe and evaluate an initial prototype of our solution that has been implemented on top of the CometCloud framework. We show how our solution is driven by the insights obtained from the data and it is able to extract data relevant to the investigators.

Pervasive computational ecosystems that combine data sources and computing/communication resources in self-managed environments, such as the ones powered by Internet of Things (IoT) devices, have the potential to automate and facilitate many aspects of our lives, and impact a variety of applications, from the management of extreme events to the optimization of everyday processes. However, this vision remains mostly unrealized despite the fact that the technology to achieve it exists, largely because of the gap between our ability to collect data and our ability to gain insight from it. In this paper, we discuss the challenges associated with providing a pervasive computational ecosystem. We then present our vision of how to best support data-driven computational ecosystems and propose a conceptual architecture that leverages ideas from software-defined environments in order to combine data, computing, and communication resources. In addition, we show how this proposed architecture enables the execution of data-driven workflows on top of these resources.

Cloud computing has generally involved the use of specialist data centres to support computation and data storage at a central site (or a limited number of sites). The motivation for this has come from the need to provide economies of scale (and subsequent reduction in cost) for supporting large scale computation for multiple user applications over (generally) a shared, multi-tenancy infrastructure. The use of such infrastructures requires moving data to a central location (data may be pre-staged to such a location prior to processing using terrestrial delivery channels and does not always require the use of a network-based transfer), undertaking processing on the data, and subsequently enabling users to download results of analysis. We extend this model using software defined networks (SDNs), whereby capability within the network can be used to support in-transit processing while data is in movement from source to destination. Using a smart building infrastructure scenario, consisting of sensors and actuators embedded within a built environment, we describe how an SDN-based architecture can be used to support real time data processing. This significantly influences the processing times to support energy optimisation of the building and reduces costs. We describe an architecture for such a distributed, multi-layered Cloud system and discuss a prototype that has been implemented using the CometCloud system, deployed across three sites in the UK and the US. Wevalidate the prototype using data from sensors within a Sports facility and making use of EnergyPlus.

Cloud engineering leverages innovations from a diverse spectrum of disciplines, from computer science and engineering to business informatics, toward the holistic treatment of key technical and business issues related to clouds.

Cloud computing continues to increase in complexity due to a number of factors: (i) increasing availability of configuration options from public Cloud providers (Amazon, for instance, offers over 4000 different configuration options), and (ii) increasing variability and types of application instances that can be deployed on such platforms, such as tuning options in hyper visors that enable different virtual machine instances to be associated with physical machines, storage, compute and I/O preferences that offer different power and price, and operating system configurations that provide differing degrees of security. This complexity can also be seen in enterprise scale data centers that dominate computing infrastructures in industry, which are growing in size and complexity, leading to complex business applications and workflows that Clouds are enabling. Autonomic computing offers self-capabilities that enable self management of systems. The underlying concepts and mechanisms of autonomics can be applied to each component within a Cloud system (resource manager/scheduler, power manager, etc.), or could be applied within an application that makes use of such a Cloud system. Understanding where such capability can be most effectively used is a decision variable often hard to fully appreciate -- an aspect explored in this tutorial.

Coordination of multiple concurrent data stream processing, carried out through a distributed Cloud infrastructure, is described. The coordination (control) is carried out through the use of a Reference net (a particular type of Petri net) based interpreter, implemented alongside the Comet Cloud system. One of the benefits of this approach is that the model can also be executed directly to support the coordination action. The proposed approach supports the simultaneous processing of data streams and enables dynamic scale-up of heterogeneous computational resources on demand, while meeting the particular quality of service requirements (throughput) for each data stream. We assume that the processing to be applied to each data stream is known a priori. The workflow interpreter monitors the arrival rate and throughput of each data stream, as a consequence of carrying out the execution using Comet Cloud. We demonstrate the use of the control strategy using two key actions - allocating and deal locating resources dynamically based on the number of tasks waiting to be executed (using a predefined threshold). However, a variety of other control actions can also be supported and are described in this work. Evaluation is carried out using a distributed Comet Cloud deployment - where the allocation of new resources can be based on a number of different criteria, such as: (i) differences between sites, i.e. Based on the types of resources supported (e.g. GPU vs. CPU only, FPGAs, etc), (ii) cost of execution, (iii) failure rate and likely resilience, etc.

Cloud computing technology is being adopted by a growing number of organizations as a way to manage, store and process data. Protecting this infrastructure and key resources such as data and recovering against cyber threats has become a critical concern. At the same time, cloud computing technology such as virtualization and on-demand provisioning can be leveraged to implement protection and recovery mechanisms for sensitive computing services. However, the energy costs of such self-healing mechanisms has been ignored. In this paper, we present a novel IDS-based self-protection mechanism at the virtual machine level inspired by software rejuvenation concepts, and a proof-of-concept prototype built upon state-of-the-art cloud technology. Our empirical evaluation aims at characterizing the cost of the proposed strategy in terms of response time and energy consumption. Preliminary results show a clear correlation between IDS accuracy, attack rate, cloud system workload, energy consumption, and response time.

Background The development of digital imaging technology is creating extraordinary levels of accuracy that provide support for improved reliability in different aspects of the image analysis, such as content-based image retrieval, image segmentation, and classification. This has dramatically increased the volume and rate at which data are generated. Together these facts make querying and sharing non-trivial and render centralized solutions unfeasible. Moreover, in many cases this data is often distributed and must be shared across multiple institutions requiring decentralized solutions. In this context, a new generation of data/information driven applications must be developed to take advantage of the national advanced cyber-infrastructure (ACI) which enable investigators to seamlessly and securely interact with information/data which is distributed across geographically disparate resources. This paper presents the development and evaluation of a novel content-based image retrieval (CBIR) framework. The methods were tested extensively using both peripheral blood smears and renal glomeruli specimens. The datasets and performance were evaluated by two pathologists to determine the concordance. Results The CBIR algorithms that were developed can reliably retrieve the candidate image patches exhibiting intensity and morphological characteristics that are most similar to a given query image. The methods described in this paper are able to reliably discriminate among subtle staining differences and spatial pattern distributions. By integrating a newly developed dual-similarity relevance feedback module into the CBIR framework, the CBIR results were improved substantially. By aggregating the computational power of high performance computing (HPC) and cloud resources, we demonstrated that the method can be successfully executed in minutes on the Cloud compared to weeks using standard computers. Conclusions In this paper, we present a set of newly developed CBIR algorithms and validate them using two different pathology applications, which are regularly evaluated in the practice of pathology. Comparative experimental results demonstrate excellent performance throughout the course of a set of systematic studies. Additionally, we present and evaluate a framework to enable the execution of these algorithms across distributed resources. We show how parallel searching of content-wise similar images in the dataset significantly reduces the overall computational time to ensure the practical utility of the proposed CBIR algorithms.

Scientific applications are moving towards dynamic and data-driven workflows with changing resource requirements. Oftentimes, these workflows combine dynamically changing resource requirements and very large computational and throughput demands. Federated computing has been explored in various contexts and has been demonstrated to be an attractive and viable model for effectively harnessing the power offered by distributed resources. However, resources themselves may also exhibit dynamic behavior with varying capabilities, capacities, and availability over time. Hence, being able to provision an appropriate computational environment at the right time is a challenging issue that cannot be solved using classic federation models where a user is presented with a fixed set of resources. In this paper we argue that an elastic execution infrastructure based on the dynamic federation of resources can help with varying application requirements. In particular, we propose a federation architecture that combines cloud abstractions with ideas from software-defined environments to offer users an integral solution for solving large scale problems in dynamically federated cyber-infrastructure.

Cloud computing is emerging as a viable platform for scientific exploration. The ideas of on-demand access to resources, "unlimited" resources as well as interesting pricing models are making scientist to move their workflows into cloud computing. However, the amount of services and different pricing models offered by the providers often overwhelm users when deciding which option is best for them. Moreover, interoperability across providers remains an open topic that forces users to develop specific solutions for each provider. In this paper, we present a service framework that enables the autonomic execution of dynamic workflows in multi-cloud environments. It also allows users to customize scheduling policies to use those resources that best fit their needs. To demonstrate the benefits of this framework, we study the execution of a real scientific workflow, with data dependencies across stages, in a multi-cloud federation using different policies and objective functions.