Search

Microservice architectures are widely used in cloud-native applications as their modularity allows for independent development and deployment of components. With the many complex interactions occurring in between components, it is difficult to determine the effects of a particular microservice rollout. Site Reliability Engineers must be able to determine with confidence whether a new rollout is at fault for a concurrent or subsequent performance problem in the system so they can quickly mitigate the issue. We present Tritium, a cross-layer analytics system that synthesizes several types of data to suggest possible causes for Service Level Objective (SLO) violations in microservice applications. It uses event data to identify new version rollouts, tracing data to build a topology graph for the cluster and determine services potentially affected by the rollout, and causal impact analysis applied to metric time-series to determine if the rollout is at fault. Tritium works based on the principle that if a rollout is not responsible for a change in an upstream or neighboring SLO metric, then the rollout's telemetry data will do a poor job predicting the behavior of that SLO metric. In this paper, we experimentally demonstrate that Tritium can accurately attribute SLO violations to downstream rollouts and outline the steps necessary to fully realize Tritium.

Modern cloud services are implemented as graphs of loosely-coupled microservices to improve programmability, reliability, and scalability. Service Level Objectives (SLOs) define end-to-end latency targets for the entire service to ensure user satisfaction. In such environments, each microservice is independently deployed and (auto-)scaled. However, it is unclear how to optimally scale individual microservices when end-to-end SLOs are violated or underutilized, and how to size each microservice to meet the end-to-end SLO at minimal total cost. In this paper, we propose Parslo---a Gradient Descent-based approach to assign partial SLOs among nodes in a microservice graph under an end-to-end latency SLO. At a high level, the Parslo algorithm breaks the end-to-end SLO budget into small incremental "SLO units", and iteratively allocates one marginal SLO unit to the best candidate microservice to achieve the highest total cost savings until the entire end-to-end SLO budget is exhausted. Parslo achieves a near-optimal solution, seeking to minimize the total cost for the entire service deployment, and is applicable to general microservice graphs that comprise patterns like dynamic branching, parallel fan-out, and microservice dependencies. Parslo reduces service deployment costs by more than 6x in real microservice-based applications, compared to a state-of-the-art partial SLO assignment scheme.

Resource management for geo-distributed infrastructures is challenging due to the scarcity and non-uniformity of edge resources, as well as the high client mobility and workload surges inherent to situation awareness applications. Due to their centralized nature, state-of-the-art schedulers that work well in datacenters lack the performance and feature requirements of such applications. We present OneEdge, a hybrid control plane that enables autonomous decision-making at edge sites for localized, rapid single-site application deployment. Edge sites handle mobility, churn, and load spikes, by cooperating with a centralized controller that allows coordinated multi-site scheduling and dynamic reconfiguration. OneEdge's scheduling decisions are driven by each application's end-to-end service level objective (E2E SLO) as well as the specific requirements of situation awareness applications. OneEdge's novel distributed state management combines autonomous decision-making at the edge sites for rapid localized resource allocations with decision-making at the central controller when multi-site application deployment is needed. Using a mix of applications on multi-region Azure instances, we show that, in contrast to centralized or fully distributed control planes, OneEdge caters to the unique requirements of situation awareness applications. Compared to a centralized control plane, OneEdge reduces deployment latency by 66% for single-site applications, without compromising E2E SLOs.

5G networks are gaining pace in development and deployment in recent years. One of 5G's key objective is to support a variety of use cases with different Service Level Objectives (SLOs). Slicing is a key part of 5G that allows operators to provide a tailored set of resources to different use cases in order to meet their SLOs. Existing works focus on slicing in the frontend or the C-RAN. However, slicing is missing in the fronthaul network that connects the frontend to the C-RAN. This leads to over-provisioning in the fronthaul and the C-RAN, and also limits the scalability of the network. In this paper, we design and implement Fronthaul Slicing Architecture (FSA), which to the best of our knowledge, is the first slicing architecture for the fronthaul network. FSA runs in the switch dataplane and uses information from the wireless schedule to identify the slice of a fronthaul data packet at line-rate. It enables multipoint-to-multipoint routing as well as packet prioritization to provide multiplexing gains in the fronthaul and the C-RAN, making the system more scalable. Our testbed evaluation using scaled-up LTE traces shows that FSA can support accurate multipoint-to-multipoint routing for 80 Gbps of fronthaul traffic. Further, the slice-aware packet scheduling enabled by FSA's packet prioritization reduces the 95th percentile Flowlet Completion Times (FCT) of latency-sensitive traffic by up to 4 times.

The rapid growth of cloud computing in cloud datacenters in recent decades greatly increases the brown energy consumption in datacenters, and hence significant increase of carbon emission that negatively impacts on the environment as well as the monetary cost. More and more cloud service providers are adopting renewable energy as the energy supply to offset the consumption of brown energy. Meanwhile, an increasing number of renewable energy generators have been built to meet the needs. However, the instability of the renewable energy cannot guarantee the support to the datacenter and the energy competition of different datacenters may lead to datacenter energy outage. In this paper, we focus on the problem of how to match different renewable energy generators to the datacenters from different cloud providers to minimize the carbon emission, monetary cost, and service level objective (SLO) violation due to renewable energy shortage. The challenges here are that the datacenters may compete in energy requesting, the renewable energy generation is not stable and the decision should be made quickly. There have been no previous efforts devoting to this problem. To solve the problem, we first test several machine learning techniques on long-term prediction accuracy on renewable energy generation and energy demand using real traces and identify SARIMA for the prediction. We then propose a multi-agent reinforcement learning based method (MARL) for each datacenter to determine how much renewable energy to request from each generator based on the predicted results. We also propose a deadline guaranteed job postponement method (DGJP) to postpone executing unurgent jobs upon insufficient renewable energy supply. The trace-driven experiments show that MARL outperforms other methods by increasing up to 35% SLO satisfaction ratio, and reducing up to 19% (0.33 billion dollars in 90 days) total monetary cost and 33% total carbon emission, and DGJP further improves the performance.

The advent of fast, byte-addressable persistent memory (PMEM) has fueled a renaissance in re-evaluating storage system design. Unfortunately, prior work has been unable to provide both consistent and fast performance because they rely on traditional cached or uncached approaches to system design, compromising at least one of the requirements. This paper presents DStore, a fast, tailless, and quiescent-free object store for non-volatile memory. To fulfill all three requirements, we propose a novel two-level approach, called DIPPER, which fully decouples the volatile frontend and persistent backend by leveraging the byte addressability and performance of PMEM. The novelty of our approach is in allowing the frontend and backend to operate independently and in parallel without affecting crash consistency. This not only avoids the need to quiesce the system but also allows for increased concurrency in the frontend through the use of observational equivalency. Using this approach, DStore achieves optimal scalability and low latency without compromising on crash consistency. Evaluation on Intel's Optane DC Persistent Memory Module (DCPMM) demonstrates that DStore can simultaneously provide fast performance, uninterrupted service, and low tail latency. Moreover, DStore can deliver up to 6x lower tail latency service level objectives (SLO) and up to 5x higher throughput SLO compared to state-of-the-art PMEM optimized systems.

Distributed machine learning has seen immense rise in popularity in recent years. Many companies and universities are utilizing computational clusters to train and run machine learning models. Unfortunately, operating such a cluster imposes large costs. It is therefore crucial to attain as high system utilization as possible. Moreover, those who offer computational clusters as a service, apart from keeping high utilization, also have to meet the required Service Level Agreements (SLAs) for the system response time. This becomes increasingly more complex in multitenant scenarios, where the time dedicated to each task has to be limited to achieve fairness. In this work, we analyze how different parameters of the machine learning job influence the response time as well as system utilization and propose Courier. Courier is a model that, based on the type of machine learning job, can select a batch size such that the response time adheres to the Service Level Objectives (SLOs) specified, while also rendering the highest possible accuracy. We gather the data by conducting real-world experiments on a BigDL cluster. Later on, we study the influence of the factors and build several predictive models which lead us to the proposed Courier model.

In cloud computing model, resource sharing introduces major benefits for improving resource utilization and total cost of ownership, but it can create technical challenges on the running performance. In practice, orchestrators are required to allocate sufficient physical resources to each Virtual Machine (VM) to meet a set of predefined performance goals. To ensure a specific service level objective, the orchestrator needs to be equipped with a dynamic tool for assigning computing resources to each VM, based on the run-time state of the target environment. To this end, we present LOOPS, a multi-loop control approach, to allocate resources to VMs based on the service level agreement (SLA) requirements and the run-time conditions. LOOPS is mainly composed of one essential unit to monitor VMs, and three control levels to allocate resources to VMs based on requests from the essential node. A tailor-made controller is proposed with each level to regulate contention among collocated VMs, to reallocate resources if required, and to migrate VMs from one host to another. The three levels work together to meet the required SLA. The experimental results have shown that the proposed approach can meet applications' performance goals by assigning the resources required by cloud-based applications.

Network Functions (NFs) perform on-path processing of network traffic. ISPs are deploying NF Virtualization (NFV) with software NFs run on commodity servers. ISPs aim to ensure that NF chains, directed acyclic graphs of NFs, do not violate Service Level Objectives (SLOs) promised by the ISP to its customers. To meet SLOs, NFV systems sometimes leverage on-path hardware (such as programmable switches and smart NICs) to accelerate NF execution. Lemur places and executes NF chains across heterogeneous hardware while meeting SLOs. Lemur's novel placement algorithm yields an SLO-satisfying NF placement while weighing many constraints: hardware memory and processing stages, server cores, link capacity, NF profiles, and NF chain interactions. Lemur's metacompiler automatically generates code and rules (in P4, Python, eBPF, C++, and OpenFlow) to stitch cross-platform NF chain execution while also optimizing resource usage. Our experiments show that Lemur is alone among competing strategies in meeting SLOs for canonical NF chains while maximizing marginal throughput (the traffic rate in excess of the service-level objective).

Video cameras have been deployed at scale today. Driven by the breakthrough in deep learning (DL), organizations that have deployed these cameras start to use DL-based techniques for live video analytics. Although existing systems aim to optimize live video analytics from a variety of perspectives, they are agnostic to the workload dynamics in real-world deployments. In this work, we present Distream, a distributed live video analytics system based on the smart camera-edge cluster architecture, that is able to adapt to the workload dynamics to achieve low-latency, high-throughput, and scalable live video analytics. The key behind the design of Distream is to adaptively balance the workloads across smart cameras and partition the workloads between cameras and the edge cluster. In doing so, Distream is able to fully utilize the compute resources at both ends to achieve optimized system performance. We evaluated Distream with 500 hours of distributed video streams from two real-world video datasets with a testbed that consists of 24 cameras and a 4-GPU edge cluster. Our results show that Distream consistently outperforms the status quo in terms of throughput, latency, and latency service level objective (SLO) miss rate.

In SaaS applications, which are mainly built following a multi-tenant architecture, the support of service variability among tenants is limited. It is due to the principle of sharing the same instance of the SaaS service between the tenants; thus, the particularities of each tenant's needs are not considered. Having to customize and adapt a SaaS service to tenants' needs and their specific contexts, while maintaining multi-tenancy, has emerged as a persisting challenge. The SaaS service adaptation has not only to be according to the functional requirements of the different tenants, but it has also to cope with the non-functional requirements in their various levels that are subject to change in time and space: i.e. within the same tenant and between tenants, respectively. For example, a tenant can just require minimal security level as it needs solely to be authenticated and has no other security concerns; while another may require a high security level and advanced and supplementary security mechanisms such as access control and encryption. There are also situations when the same tenant changes its desired quality level through time; for instance, when there is an increase of service requests in a critical period, the tenant may want to decrease the response time and to increase the uptime value in order to smoothly perform his requests. Therefore, the application needs to be adapted dynamically in order to meet every tenant's quality requirements as if he is the only one to consume the service.Software Product Lines Engineering (SPLE) has largely tackled the variability management field in the literature. It enables high reusability in shorter time, at lower costs and with higher quality by creating product families or lines that share commonalities and have variation points. The derivation of the final product is performed at the level of those variation points while considering the inter-and intra- dependencies between the artifacts of a product line.Our proposed approach is based on the SPLE principles to build SaaS services tailored to tenant-specific Service Level Agreements (SLAs). We defined the Domain Engineering phase to build SLA families. Based on domain analysis, the non-functional requirements of the tenants, their commonalities, variation points and variants are captured and modeled. A Generic SLA is generated which contains the terms and the Service Level Objectives (SLOs) of all the contracting tenants. From this Generic SLA, the proposed middleware generates core and tenant-specific policies. The Configurator, a middleware component, adapts the application according to these defined policies based on annotations in the form of key-value pairs.As contributions of our article [1]: we proposed the architecture of the proposed middleware and its different components; we defined the Policy metamodel, that models policies retrieved from the Generic SLA as annotations, and described the model-to-model transformations from VariableSLA metamodel to Policy metamodel; and we proposeed a Configurator metamodel that models and describes how the Configurator component of the middleware configures the SaaS service.Our approach differs from the literature in that it enables building a family of SLAs: a generic SLA and tenant-specific policies; linking, thus, the non-functional variability with the SLA. Moreover, it is not limited to a specific quality attribute but supports any quality attribute expressed in the SLA. It allows configurations at design time and at runtime through annotating the SaaS service components with adequate quality levels.

The increasing demand for cloud-based inference services requires the use of Graphics Processing Unit (GPU). It is highly desirable to utilize GPU efficiently by multiplexing different inference tasks on the GPU. Batched processing, CUDA streams and Multi-process-service (MPS) help. However, we find that these are not adequate for achieving scalability by efficiently utilizing GPUs, and do not guarantee predictable performance.GSLICE addresses these challenges by incorporating a dynamic GPU resource allocation and management framework to maximize performance and resource utilization. We virtualize the GPU by apportioning the GPU resources across different Inference Functions (IFs), thus providing isolation and guaranteeing performance. We develop self-learning and adaptive GPU resource allocation and batching schemes that account for network traffic characteristics, while also keeping inference latencies below service level objectives. GSLICE adapts quickly to the streaming data's workload intensity and the variability of GPU processing costs. GSLICE provides scalability of the GPU for IF processing through efficient and controlled spatial multiplexing, coupled with a GPU resource re-allocation scheme with near-zero (

Incident routing is critical for maintaining service level objectives in the cloud: the time-to-diagnosis can increase by 10x due to mis-routings. Properly routing incidents is challenging because of the complexity of today's data center (DC) applications and their dependencies. For instance, an application running on a VM might rely on a functioning host-server, remote-storage service, and virtual and physical network components. It is hard for any one team, rule-based system, or even machine learning solution to fully learn the complexity and solve the incident routing problem. We propose a different approach using per-team Scouts. Each teams' Scout acts as its gate-keeper --- it routes relevant incidents to the team and routes-away unrelated ones. We solve the problem through a collection of these Scouts. Our PhyNet Scout alone --- currently deployed in production --- reduces the time-to-mitigation of 65% of mis-routed incidents in our dataset.

Bing's monetization pipeline is one of the largest and most critical streaming workloads deployed in Microsoft's internal data lake. The pipeline runs 24/7 at a scale of 3500 YARN containers and is required to meet a Service Level Objective (SLO) of low tail latency. In this paper, we highlight some of the unique challenges imposed by this large scale of operation: other concurrent workloads sharing the cluster may cause random performance deterioration; unavailability of external dependencies may cause temporary stalls in the pipeline; scarcity in the underlying resource manager may cause arbitrarily long delays or rejection of container allocation requests. Weathering these challenges requires specially tailored dynamic control policies that react to these issues as and when they arise. We focus on the problem of reducing the latency in the tail, i.e., 99th percentile (p99), by detecting and mitigating slow instances through speculative replication. We show that widely used approaches do not satisfactorily solve this issue at our scale. A conservative approach is hesitant to acquire additional resources, reacts too slowly to the changes in the environment and therefore achieves little improvement in p99 latency. On the other hand, an aggressive approach overwhelms the underlying resource manager with unnecessary resource requests and paradoxically worsens the p99 latency. Our proposed approach, Spur, is designed for this challenging environment. It combines aggressive detection of slow instances with smart pruning of false positives to achieve a far better trade-off between these conflicting objectives. Using only 0.5% additional resources (similar to the conservative approach), we demonstrate a 10% -38% improvement in the tail latency compared to both conservative and aggressive approaches.

SaaS application instances typically serve multiple tenants to improve cost-efficiency. This results in the need for adaptive performance isolation between tenants in order to guarantee custom service level objectives (SLOs) about request latency or throughput. Current solutions, which are based on request scheduling algorithms, suffer from SLO instability under globally varying workloads. This means that the configuration for an SLO has to be recalibrated when total workload patterns change such as an increase or decrease in the number of subscribed tenants, or the application becomes co-located with other types of resource-intensive applications. Lately container technology such as Docker and container orchestration frameworks like Kubernetes have been used to increase cost-efficiency, multi-tenancy and elasticity. This paper investigates if the problem of adaptive performance isolation can be mapped to resource management concepts of Kubernetes through a series of experiments. These experiments show that Kubernetes provides good support for QoS differentiation and adaptive resource allocation by grouping tenants according to their SLO class (e.g gold vs bronze) in different containers. Moreover, SLO instability does not occur when co-locating these containers with other container-based applications provided that a few interferences between CPU-, memory- and disk-io intensive applications are taken into account. However SLO instability does occur when the number of subscribed tenants changes. This latter problem is not caused by the replication and auto-scaling concepts of Kubernetes, but by a non-linear resource scaling phenomenon that is inherent when the goal is to meet multiple custom SLOs in a cost-optimal way.

During the past few years, all leading cloud providers introduced burstable instances that can sprint their performance for a limited period to address sudden workload variations. Despite the availability of burstable instances, there is no clear understanding of how to minimize the waste of resources by regulating their burst capacity to the workload requirements. This is especially true when it comes to non-CPU-intensive applications. In this paper, we investigate how to limit network and I/O usage to optimize the efficiency of the bursting process. We also study which resource shall be controlled to benefit both cloud providers and end-users. We design MRburst (Multi-Resource burstable performance scheduler) to automatically limit multiple resources (i.e., network, I/O, and CPU) and make the application comply with a user-defined service level objective (SLO) while minimizing wasted resources. MRburst is evaluated on Amazon EC2 using two multi-resource applications: an FTP server and a Ceph system. Experimental results show that MRburst outperforms state-of-the-art approaches by allowing instances to speed up their performance for up to 2.4 times longer period while meeting SLO.

Cloud customers want strong, understandable promises (Service Level Objectives, or SLOs) that their applications will run reliably and with adequate performance, but cloud providers don't want to offer them, because they are technically hard to meet in the face of arbitrary customer behavior and the hidden interactions brought about by statistical multiplexing of shared resources. Existing cloud SLOs are more concerned with defending against corner cases than defining normal behavior. This and other tensions make SLOs surprisingly hard to define. We show that this problem shares some similarities with the challenges of applying statistics to make decisions based on sampled data. We argue that a mutually beneficial set of Service Level Expectations (SLEs) and Customer Behavior Expectations (CBEs) ameliorates many of the problems of today's SLOs by explicitly sharing risk between customer and service provider.

Microservices enable a fine-grained control over the cloud applications that they constitute and thus became widely-used in the industry. Each microservice implements its own functionality and communicates with other microservices through language- and platform-agnostic API. The resources usage of microservices varies depending on the implemented functionality and the workload. Continuously increasing load or a sudden load spike may yield a violation of a service level objective (SLO). To characterize the behavior of a microservice application which is appropriate for the user, we define a MicroService Capacity (MSC) as a maximal rate of requests that can be served without violating SLO.The paper addresses the challenge of identifying MSC individually for each microservice. Finding individual capacities of microservices ensures the flexibility of the capacity planning for an application. This challenge is addressed by sandboxing a microservice and building its performance model. This approach was implemented in a tool Terminus. The tool estimates the capacity of a microservice on different deployment configurations by conducting a limited set of load tests followed by fitting an appropriate regression model to the acquired performance data. The evaluation of the microservice performance models on microservices of four different applications shown relatively accurate predictions with mean absolute percentage error (MAPE) less than 10%.The results of the proposed performance modeling for individual microservices are deemed as a major input for the microservice application performance modeling.

With increase in cellular-enabled IoT devices having diverse traffic characteristics and service level objectives (SLOs), handling the control traffic in a scalable and resource-efficient manner in the cellular packet core network is critical. The traditional monolithic design of the cellular core adopted by service-providers is inflexible with respect to the diverse requirements and bursty loads of IoT devices, specifically for properties such as elasticity, customizability, and scalability. To address this key challenge, we focus on the most critical control plane component of the cellular packet core network, the Mobility Management Entity (MME). We present MMLite, a functionally decomposed and stateless MME design wherein individual control procedures are implemented as microservices and states are decoupled from their processing, thus enabling elasticity and fault tolerance. For SLO compliance, we develop a multi-level load balancing approach based on skewed consistent hashing to efficiently distribute incoming connections. We evaluate the performance benefits of MMLite over existing approaches with respect to scaling, fault tolerance, SLO compliance and resource efficiency.

We present Henge, a system to support intent-based multi-tenancy in modern stream processing applications. Henge supports multi-tenancy as a first-class citizen: everyone inside an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Additionally, Henge allows each tenant (job) to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput. In a multi-tenant cluster, the Henge scheduler adapts continually to meet jobs' SLOs in spite of limited cluster resources, and under dynamic input workloads. SLOs are soft and are based on utility functions. Henge continually tracks SLO satisfaction, and when jobs miss their SLOs, it wisely navigates the state space to perform resource allocations in real time, maximizing total system utility achieved by all jobs in the system. Henge is integrated in Apache Storm and we present experimental results using both production topologies and real datasets.

With the rise of machine learning in production, we need to talk about operational data science. The talk introduces a pragmatic method on how to measure the response quality of a recommendation service. To that end the definition of a successful response is introduced and guidelines how to capture the rate of successful responses are presented. There are several changes that can happen during the serving phase of a model which negatively affect the quality of the algorithmic response. A few examples are: • The model is updated and the new version is inferior to the previous one. • The latest deployment of the stack that processes the request and serves the model contains a bug. • Changes in the infrastructure lead to performance loss. An example in an e-commerce setting is switching to a different microservice to obtain article metadata used for filtering the recommendations. • The input data changes. Typical reasons might be a client application that releases a bug (e.g., lowercasing a case sensitive identifier) or changes a feature in a way that affects the data distribution such as allowing all users to use the product cart instead of previously allowing it only for logged in users. If the change is not detected training data and serving data diverge. Current monitoring solutions mostly focus on the completion of a request without errors and the request latency. That means the mentioned examples would be hard to detect despite the response quality being significantly degraded, sometimes permanently. In addition to not being able to detect the mentioned changes, it can be argued that current monitoring practices are not sufficient to capture the performance of a recommender system or any other data driven service in a meaningful way. We might for instance have returned popular articles as a fallback in a case where personalized recommendations were requested. We should record that response as unsuccessful. A new paradigm for measuring response quality should fulfil the following criteria: • comparable across models • simple and understandable metrics • measurements are collected in real time • allows for actionable alerting on problems The response quality is defined as an approximation of how well the response fits the defined business and modelling case. The goal is to bridge the gap between metrics used during model learning and technical monitoring metrics. Ideally we would like to obtain Service Level Objectives (SLO)[1] that contain this quality aspect and can be discussed with the different client applications based on the business cases, e.g., "85% of the order confirmation emails contain personalized recommendations based on the purchase." A case study will illustrate how algorithmic monitoring was introduced in the recommendation team at Zalando. Zalando is one of Europe's largest fashion retailers and multiple recommendation algorithms serve many online and offline use cases. You will see several examples of how the monitoring helped to identify bugs or diagnose quality problems.

Modern resource management frameworks guarantee low tail latency for long-running services using the resource over-provisioning method, resulting in serious waste of resource and increasing the service costs greatly. To reduce the over-provisioning cost, we present EFRA, an elastic and fine-grained resource allocator that enables much more efficient resource provisioning while guaranteeing the tail latency Service Level Objective (SLO). EFRA achieves this through the cooperation of three key components running on a containerized platform: The period detector identifies the period features of the workload through a convolution-based time series analysis. The resource reservation component estimates the just-right amount of resources based on the period analysis through a top-K based collaborative filtering approach. The online reprovisioning component dynamically adjusts the resources for further enforcing the tail latency SLO. Testbed experiments show that EFRA is able to increase the average resource utilization to 43%, and save up to 66% resources while guaranteeing the same tail latency objective.

Resource flexing is the notion of allocating resources on-demand as workload changes. This is a key advantage of Virtualized Network Functions (VNFs) over their non-virtualized counterparts. However, it is difficult to balance the timeliness and resource efficiency when making resource flexing decisions due to unpredictable workloads and complex VNF processing logic. In this work, we propose an Elastic resource flexing system for Network functions VIrtualization (ENVI) that leverages a combination of VNF-level features and infrastructure-level features to construct a neural-network-based scaling decision engine for generating timely scaling decisions. To adapt to dynamic workloads, we design a window-based rewinding mechanism to update the neural network with emerging workload patterns and make accurate decisions in real time. Our experimental results for real VNFs (IDS Suricata and caching proxy Squid) using workloads generated based on real-world traces, show that ENVI provisions significantly fewer (up to 26%) resources without violating service level objectives, compared to commonly used rule-based scaling policies.

The workflows of the predominant user-facing datacenter services, including web searching and social networking, are underlaid by various Fork-Join structures. Due to the lack of understanding the performance of Fork-Join structures in general, today's datacenters often resort to resource overprovisioning, operating under low resource utilization, to meet stringent tail-latency service level objectives (SLOs) for such services. Hence, to achieve high resource utilization, while meeting stringent tail-latency SLOs, it is of paramount importance to be able to accurately predict the tail latency for a broad range of Fork-Join structures of practical interests.In this paper, we propose ForkTail, a black-box Fork-Join tail latency prediction model that covers a wide range of Fork-Join structures. In ForkTail, all Fork nodes are treated as black boxes, admitting both homogeneous and inhomogeneous cases, and different requests in the request flow are allowed to spawn different numbers of tasks forked to different numbers of Fork nodes. On the basis of the central limit theorem for queuing models under heavy load, we are able to arrive at a highly computational effective, empirical expression for the tail latency as a function of the means and variances of the task response times. Since this expression can be applied to request sub-flows at any granularities, it can be used for tail-latency prediction for services in a consolidated environment, where different services and applications may share the same datacenter cluster resources. Our extensive testing results based on model-based and trace-driven simulations, as well as a real-world case study in a cloud environment demonstrate that the expression can consistently predict the tail latency within 20% and 15% prediction errors at 80% and 90% load levels, respectively. Moreover, our sensitivity analysis demonstrates that such errors can be well compensated for with no more than 5% and 3% resource overprovisioning at these two load levels, respectively. This, together with its extremely low computational complexity, makes ForkTail a viable tool for both offline and online job scheduling and resource provisioning for user-facing datacenter applications.

Computational sprinting speeds up query execution by increasing power usage for short bursts. Sprinting policy decides when and how long to sprint. Poor policies inflate response time significantly. We propose a model-driven approach that chooses between sprinting policies based on their expected response time. However, sprinting alters query executions at runtime, creating a complex dependency between queuing and processing time. Our performance modeling approach employs offline profiling, machine learning, and first-principles simulation. Collectively, these modeling techniques capture the effects of sprinting on response time. We validated our modeling approach with 3 sprinting mechanisms across 9 workloads. Our performance modeling approach predicted response time with median error below 4% in most tests and median error of 11% in the worst case. We demonstrated model-driven sprinting for cloud providers seeking to colocate multiple workloads on AWS Burstable Instances while meeting service level objectives. Model-driven sprinting uncovered policies that achieved response time goals, allowing more workloads to colocate on a node. Compared to AWS Burstable policies, our approach increased revenue per node by 1.6X.

Spark Streaming discretizes streams of data into micro-batches, each of which is further sub-divided into tasks and processed in parallel to improve job throughput. Previous work [2, 3] has lowered end-to-end latency in Spark Streaming. However, two causes of high tail latencies remain unaddressed: 1) data is not load-balanced across tasks, and 2) straggler tasks can increase end-to-end latency by 8 times more than the median task on a production cluster [1]. We propose a feedback-control mechanism that allows frameworks to adaptively load-balance workloads across tasks according to their processing speeds. The task runtimes are thus equalized, lowering end-to-end tail latency. Further, this reduces load on machines that have transient resource bottlenecks, thus resolving the bottlenecks and preventing them from having an enduring impact on task runtimes.

In a cloud data center, a single physical machine simultaneously executes dozens of highly heterogeneous tasks. Such colocation results in more efficient utilization of machines, but, when tasks' requirements exceed available resources, some of the tasks might be throttled down or preempted. We analyze version 2.1 of the Google cluster trace that shows short-term (1 second) task CPU usage. Contrary to the assumptions taken by many theoretical studies, we demonstrate that the empirical distributions do not follow any single distribution. However, high percentiles of the total processor usage (summed over at least 10 tasks) can be reasonably estimated by the Gaussian distribution. We use this result for a probabilistic fit test, called the Gaussian Percentile Approximation (GPA), for standard bin-packing algorithms. To check whether a new task will fit into a machine, GPA checks whether the resulting distribution's percentile corresponding to the requested service level objective, SLO is still below the machine's capacity. In our simulation experiments, GPA resulted in colocations exceeding the machines' capacity with a frequency similar to the requested SLO., Comment: Author's version of a paper published in ACM SoCC'17

Service providers want to reduce datacenter costs by consolidating workloads onto fewer servers. At the same time, customers have performance goals, such as meeting tail latency Service Level Objectives (SLOs). Consolidating workloads while meeting tail latency goals is challenging, especially since workloads in production environments are often bursty. To limit the congestion when consolidating workloads, customers and service providers often agree upon rate limits. Ideally, rate limits are chosen to maximize the number of workloads that can be co-located while meeting each workload's SLO. In reality, neither the service provider nor customer knows how to choose rate limits. Customers end up selecting rate limits on their own in some ad hoc fashion, and service providers are left to optimize given the chosen rate limits. This paper describes WorkloadCompactor, a new system that uses workload traces to automatically choose rate limits simultaneously with selecting onto which server to place workloads. Our system meets customer tail latency SLOs while minimizing datacenter resource costs. Our experiments show that by optimizing the choice of rate limits, WorkloadCompactor reduces the number of required servers by 30--60% as compared to state-of-the-art approaches.

In recent years, stream processing systems have been deployed in almost every organization due to the explosion of large-scale analytics applications. Our discussions with users of these systems within Microsoft and Twitter have revealed that a major challenge with these frameworks is to tune them in order to meet the required performance and also maintain this level of performance over time. In this paper, we present the open problems and challenges in supporting streaming systems that self-regulate. Such systems automatically adjust their configuration to meet service level objectives (SLOs) even in the presence of external load variations or internal faults such as slow hardware. To address some of these challenges, we propose using machine learning techniques such as supervised learning and reinforcement learning which can potentially further improve the application management lifecycle. We believe that exploring machine learning in the context of self-regulating streaming systems is a rich area for future research with can impact the ways streaming applications are managed.

In this competitive environment, service provider organizations are keen to create value for their client organizations and differentiate their services during delivery of IT services outsourcing projects. However, there are variations in value delivery as per client expectations. It is critical to understand how alignment can be achieved between B-to-B client and vendor organizations to minimize this variation. Through dyadic case studies, this research paper develops a framework of (mis)aligned practices at three levels of conceptualization- contractual alignment, knowledge alignment and business alignment. Further, a project alignment measurement framework has been developed along with propositions for future study. This research adopted a practice based analyses of organizations Findings from data collected through interviews and project documentations resulted in nineteen practices showing aligned behaviors and twenty-three practices showing mis-aligned behaviors at contractual alignment level; six practices showing aligned behaviors and eleven practices showing mis-aligned behaviors at knowledge alignment level; and fourteen practices showing aligned behaviors and eleven practices showing mis-aligned behaviors at business alignment level

The concept of service level agreements (SLAs) defines the idea of a reliable contract between service providers and their users. SLAs provide information on the scope, the quality and the responsibilities of a service and its provider. Service level objectives (SLOs) define the detailed, measurable conditions of the SLAs. After service deployment, SLAs are monitored for situations, that lead to SLA violations. However, the SLA monitoring infrastructure is usually specific to the underlying system infrastructure, lacks generalization, and is often limited to measurement-based approaches. This makes it hard to apply the results from SLA monitoring in other stages of the software life-cycle. In this paper we propose the mapping of concerns defined in SLAs to the performance metrics queries using the Descartes Query Language (DQL). The benefit of our approach is that the same performance query can then be reused for evaluation of performance concerns throughout the entire life-cycle, and regardless of which approach is used for evaluation.

The present paper has been designed as a systematic descriptive study to explore the service quality of Common Service Centers (CSC's) in India through a conceptual model.

Cloud resource usage prediction is an important pre-requisite for optimal scheduling and load balancing. It is a very challenging task as a number of users with varied resource requests enter and leave the cloud network dynamically. Predicting resource usage in advance can aid service providers in better capacity planning to meet their service level objectives. In this paper, we propose a fractional differencing based method to capture long range dependence in time series data. The proposed model is evaluated on a Google cluster trace. Empirical results show that fractionally differencing the data gives better results as compared to non-fractionally differenced data. To take advantage of existing models, a hybrid method for resource usage prediction is proposed which combines the predictions of existing models to generate better forecasts.

Cloud computing provides on-demand resources with no constraints of physical locations. It allows customers to save upfront infrastructure costs and focus on features that discriminate their core businesses. The increasing number of offered services makes manual selection of the most suitable service for an application deployment time-consuming and challenging. It also requires a high level of user expertise to make proper decisions. In this paper, we present QuARAM Service Recommender platform, a self-adaptive Infrastructure-as-a-Service (IaaS) service selection system that recommends a list of suitable services for cloud application deployment based on application requirements and customer preferences. The process begins with automatic extraction of the application's features, requirements and customer preferences and provides a list of potential services for the application deployment (i.e., resource allocation in our context). Initial experiments show promising results for up to 90% precision of recommended services.

Information service is an important assistive tool in the knowledge sharing process. The knowledge sharing process covers a wide range of operations ranging from a course registration process to flight ticket buying. A good service could help both the service providers and the service users. The service users could get the right information at the right time and at the right place; whereas the service providers could reduce the amount of work created by uninformed users. In this paper, an electronic service framework (e-service) is proposed to assist the information service. This framework is based on an information retrieval technique to discover information. As a result, the information can be accessed at any place and any time. Furthermore, the issue of information quality is also a part of this framework. Each piece of information will be provided with tags and a weight. The tags will be used for categorization while the weight will be used to determine the quality of the information. Ultimately, the information will be evaluated by the users whose comments and scores will later be used to improve the system.

Formed in 2012 after NC State "went Google," the Google Service Team (GST), made up of staff from different units in the Office of Information Technology as well as representatives from across campus, has thrived and meets on a regular, most often weekly, basis. This service team model has been successful in running an enterprise-wide cloud service, in part because of a clear mission and solid representation from outside of the central IT organization. Additionally, service ownership has given a single point of contact and continuity in decision-making. The GST also helps streamline requests, issues, and escalations across teams as IT staff know where and how to funnel these items.This paper and presentation will discuss NC State's Google Service Team (GST), including its mission and structure (including ties into governance) and the challenges and rewards of running and managing an enterprise-wide cloud service for both the team members and customer base.

This paper investigates an on-demand spatial service broker for suggesting service provider propositions and the corresponding estimated waiting times to mobile consumers while meeting the consumer's maximum travel distance and waiting time constraints. The goal of the broker is to maximize the number of matched requests while also keeping the "ecosystem" functioning by engaging many service providers and balancing their assigned requests to provide them with incentives to stay in the system. This problem is important because of its many related societal applications in the on-demand and sharing economy (e.g. on-demand ride hailing services, on-demand food delivery, etc). Challenges of this problem include the need to satisfy many conflicting requirements for the broker, consumers and service providers and the high computational complexity for a large number of consumers and service providers. Related work in spatial crowdsourcing and ridesharing has mainly focused on maximizing the number of matched requests and minimizing travel cost, but did not consider the importance of engaging more service providers and balancing their assignments, which could become a priority when the available supply exceeds the demand. In this work, we propose a new category of service provider centric heuristics for meeting these conflicting requirements. We evaluated our algorithms using synthetic datasets with real-world characteristics. Experimental results show that our proposed heuristics can achieve a larger number of matched requests when supply and demand are balanced. They also engage a larger number of service providers with a more balanced provider assignment when the available supply greatly exceeds demand.

Meeting tail latency Service Level Objectives (SLOs) in shared cloud networks is both important and challenging. One primary challenge is determining limits on the multi-tenancy such that SLOs are met. Doing so involves estimating latency, which is difficult, especially when tenants exhibit bursty behavior as is common in production environments. Nevertheless, recent papers in the past two years (Silo, QJump, and PriorityMeister) show techniques for calculating latency based on a branch of mathematical modeling called Deterministic Network Calculus (DNC). The DNC theory is designed for adversarial worst-case conditions, which is sometimes necessary, but is often overly conservative. Typical tenants do not require strict worst-case guarantees, but are only looking for SLOs at lower percentiles (e.g., 99th, 99.9th). This paper describes SNC-Meister, a new admission control system for tail latency SLOs. SNC-Meister improves upon the state-of-the-art DNC-based systems by using a new theory, Stochastic Network Calculus (SNC), which is designed for tail latency percentiles. Focusing on tail latency percentiles, rather than the adversarial worst-case DNC latency, allows SNC-Meister to pack together many more tenants: in experiments with production traces, SNC-Meister supports 75% more tenants than the state-of-the-art.

Generally web service selection is performed based on nonfunctional (Quality of Service) attributes. But Quality of Service (QoS) attribute values may not be equal to the published values by the service providers. Therefore, the users want to concern about the trustworthiness of the services or service providers. The trustworthiness of a service is measured by the users' ratings. There may be the dishonest users who provide the fake ratings for commercial benefit. These users are called malicious or collusive users and can be treated like outliers. Now, we have proposed an approach to identify these outliers using data mining approach. Then Composite Web Service Selection is performed using Genetic Algorithms (GA). The problem is to find an optimal concrete service composition plan under users' QoS constraints. The QoS-aware composite web service selection is NP-hard problem. But it can be mapped with Multi-dimensional Multi-choice Knapsack Problem(MMKP), which is NP-complete[6]. Therefore, we use Genetic Algorithm(GA) to find the near-optimal composite service plan with QoS constraints.

Service discovery in global software markets is performed by brokers who act as intermediaries between service requesters and service providers. In order to discover services, brokers apply service matching for determining whether the specification of a provided service satisfies the requester's requirements. Brokers can already choose between a lot of different service matching approaches considering different service properties (structural, behavioral, and non-functional properties). Different matching approaches can be combined into configurable matching processes leading to a high matching quality (e.g., accurate matching results). However, this combination and configuration is a manual procedure and has to be repeated for different requesters' or market requirements regarding matching quality. In this paper, we propose our framework MatchBox, which supports a broker in reusing existing matching approaches and combining them in a model-driven way based on a reconfigurable model of the matching process. Using this reconfigurable model, MatchBox takes care of control and data flow between matching approaches and executes the modeled processes automatically. As a case study, we integrated eleven matchers into MatchBox to demonstrate that it remains flexibility and reduces effort for a broker at the same time.

Cloud-based software systems are expected to deliver reliable performance under dynamic workload while efficiently managing resources. Conventional monitoring frameworks provide limited support for flexible and intuitive performance queries. In this paper, we present a prototype monitoring and control platform for clouds that is a better fit to the characteristics of cloud computing (e.g. extensible, user-defined, scalable). Service Level Objectives (SLOs) are expressed graphically as Performance Trees, while violated SLOs trigger mitigating control actions.

The increase in internet coverage and decrease in internet access price has resulted in demand for good internet service. Clients want some guarantee in internet access quality. In this paper, we present a model in which clients are guaranteed connection and bandwidth and if clients do not get the service they request, the service provider pays a penalty to the clients. We consider a system of internet clients with multiple internet service provider (ISP) connections to a set of ISPs. When a client arrives, an ISP has to decide whether to accept the client, and then the price to charge from the client for the duration of its connection. Rejection of a client results in a penalty and delay in getting the requested bandwidth also incurs a penalty. We assume a Poisson arrival process with the rate of arrival sensitive to the price being charged. A client requests bandwidth for a time that is exponentially distributed, then the client is idle for a time that is also exponentially distributed; and then either the client departs or requests bandwidth again after the idle period is over. A service provider tries to maximize its income by charging appropriate prices based on its current state and deciding whether to accept more clients or not. Since penalties are imposed, such solutions also automatically balance load among service providers, and so the quality of service to clients improves. We present solutions that maximize the income of service providers. The solutions are then compared using simulation. Simulation results show that our analysed solution significantly improve quality of service of clients and increase the income of service providers as compared to a simple heuristic based solution that is otherwise could to be used.

As the multitude and complexity of cloud services increases, the role of cloud brokers in the cloud service ecosystems becomes increasingly important. In particular, the lack of standard mechanisms that allow for the comparison of cloud service specifications against user requirements taking into account the implicit uncertainty and vagueness is a major hindrance during the cloud service evaluation and selection. In this paper, we discuss the Preference-based cLoud Service Recommender (PuLSaR) that uses a holistic multi-criteria decision making (MCDM) approach for offering optimisation as brokerage service. The specification and implementation details of this dedicated software are thoroughly discussed while the background method used is summarised. Both method and brokerage service allow for the multi-objective assessment of cloud services in a unified way, taking into account precise and imprecise metrics and dealing with their fuzziness.

The elasticity promised by cloud computing does not come for free. Providers need to reserve resources to allow users to scale on demand, and cope with workload variations, which results in low utilization. The current response to this low utilization is to re-sell unused resources with no Service Level Objectives (SLOs) for availability. In this paper, we show how to make some of these reclaimable resources more valuable by providing strong, long-term availability SLOs for them. These SLOs are based on forecasts of how many resources will remain unused during multi-month periods, so users can do capacity planning for their long-running services. By using confidence levels for the predictions, we give service providers control over the risk of violating the availability SLOs, and allow them trade increased risk for more resources to make available. We evaluated our approach using 45 months of workload data from 6 production clusters at Google, and show that 6--17% of the resources can be re-offered with a long-term availability of 98.9% or better. A conservative analysis shows that doing so may increase the profitability of selling reclaimed resources by 22--60%.

Meeting service level objectives (SLOs) for tail latency is an important and challenging open problem in cloud computing infrastructures. The challenges are exacerbated by burstiness in the workloads. This paper describes PriorityMeister -- a system that employs a combination of per-workload priorities and rate limits to provide tail latency QoS for shared networked storage, even with bursty workloads. PriorityMeister automatically and proactively configures workload priorities and rate limits across multiple stages (e.g., a shared storage stage followed by a shared network stage) to meet end-to-end tail latency SLOs. In real system experiments and under production trace workloads, PriorityMeister outperforms most recent reactive request scheduling approaches, with more workloads satisfying latency SLOs at higher latency percentiles. PriorityMeister is also robust to mis-estimation of underlying storage device performance and contains the effect of misbehaving workloads.

With the wide popularity of Cloud Computing, Service-oriented Computing is becoming the de-facto approach for the development of distributed systems. This has introduced the issue of trustworthiness with respect to the services being provided. Service Requesters are provided with a wide range of services that they can select from. Usually the service requester compare between these services according to their cost and quality. One essential part of the quality of a service is the trustworthiness properties of such services. Traditional service models focuses on service functionalities and cost when defining services. This paper introduces a new service model that extends traditional service models to support trustworthiness properties.

In the past decade, the IT industry has experienced a paradigm shift as computing resources became available as a utility through cloud based services. In spite of the wider adoption of cloud computing platforms, some businesses and organizations hesitate to move all their applications to the cloud due to performance concerns. Existing practices in application performance management rely heavily on white-box modeling and diagnosis approaches or on performance troubleshooting "cookbooks" to find potential bottlenecks and remediation steps. However, the scalability and adaptivity of such approaches remain severely constrained, especially in a highly-dynamic, consolidated cloud environment. For performance isolation and differentiation, most modern hypervisors offer powerful resource control primitives such as reservations, limits, and shares for individual virtual machines (VMs). Even so, with the exploding growth of virtual machine sprawl, setting these controls properly such that co-located virtualized applications get enough resources to meet their respective service level objectives (SLOs) becomes a nearly insoluble task. These challenges present unique opportunities in leveraging the rich telemetry collected from applications and systems in the cloud, and in applying statistical learning, optimization, and control based techniques to developing model-based, automated application performance management frameworks. There has been a large body of research in this area in the last several years, but many problems remain. In this talk, I'll highlight some of the automated and data-driven performance management techniques we have developed, along with related technical challenges. I'll then discuss open research problems, in hope to attract more innovative ideas and solutions from a larger community of researchers and practitioners.

Currently several enterprises depend on services provided to them by other parties for the realization of their own service offer. The interconnections in terms of services offered and required by companies shape complex systems, called service networks. The new cloud-computing paradigm is stressing the concept of the governance of relationship between IT customers and providers. The relationship is based on Service Level Agreements (SLA) and obligations, described inside contracts between providers and customers. Every customer needs to agree with a SLA in order to lease a new service. SLAs describe provisioning terms and encapsulate QoS characteristics as well as functional properties. Traditionally, providers define SLAs in which they guarantee explicit provisioning service level bounds for an agreed period. In the scientific literature, SLAs are hardly viewed as end-user documents, and merely as automated processes that assist the monitoring and scheduling of resources. In contrast, cloud IT marketplaces treat SLAs as static documents that do not allow for any processing. Moreover, strong diversity exists in how service providers coming from distinct business and socio-economical domains formulate and exercise their provisioning responsibilities. Such issues need to be resolved in order to make SLAs enacting tools for managers and services' providers in order to govern contracts. The contribution of this paper is to discuss some relevant open issues not covered by current contract management approaches and tools. In particular we will explore how service composition, with specific focus on service levels, limits and shapes technical, legal and organizational aspects related to the contract of the composed service.

Integrating a distributed software system using publically available software services saves effort, time, and cost. One key step in this process is the service selection which identifies a relevant set of services for this integration. In an open service marketplace, it is hard to judge the trust of software services using a static view (consisting of service specifications) published by their developers. Instead, the concepts of trust in the context of services needs to be properly quantified, specified, negotiated, and then used in the selection process. Prevalent service selection and negotiation approaches do not consider the trust aspect of services. Trustworthy service representation, selection, and negotiation are challenging tasks due to the subjective and temporal nature of trust, lack of standards, and associated uncertainty. This proposal defines the trust of a service, quantifies the trust by monitoring and aggregating various evidences, represents the trust in the service specification and improves the service selection and negotiation process using this representation. Publically available software objects (from Java collection framework library) and mobile app services (from the Android marketplace) are used as datasets to empirically evaluate this proposal.

Cloud computing encourages application to migrate into it for economic of scale, where they rent shared resources to deliver services. Service Level Agreements(SLA) plays an important role in assisting various applications providing high-quality services to end users in cloud's complex and uncertain environments. Most of the existing work tries to support application claimed quality by help cloud make decisions of computing resources allocation during runtime. In this paper, we propose an approach for applications to maintain quality requirements by runtime cache adjustment in consideration of service level objectives (SLOs) and unpredictable workload in cloud, which can be taken as a complement to the existing work. Our approach includes application SLO modeling and mapping to monitor metrics during runtime, and an algorithm to adapting caches according to runtime status and SLOs. The approach has been applied to a real-world SNS application which proves effectiveness of our approach.