Search

We explore settings where it is necessary (due to physical or operational constraints) or desirable (due to synergies or ease of implementation) to assign resources to tasks in a synchronous manner. We model the system as a queueing network with flexible servers and introduce the notion of a configuration to address the synchronous assignment of servers. This allows for a unified approach to determine the effects of resource synchronization, covering a wide range of problems in the literature. The maximal capacity of the system is given by the solution of a linear programming problem that also provides the optimal fractions of time the servers should spend in different configurations. This is used as a basis for constructing policies that have capacity arbitrarily close to the maximal capacity. We contrast synchronous server assignment with an asynchronous approach (focusing on independently scheduling individual servers rather than configurations) and show that synchronous server assignment is attractive with respect to applicability (it can capture constraints on server assignment and synergies among servers), implementation (it may have significantly fewer combinations of server allocations), and capacity (when both are applicable, asynchronous and synchronous server assignment will yield the same maximal capacity). Finally, we illustrate our modeling framework using several examples. [ABSTRACT FROM AUTHOR]

Scheduling of MapReduce jobs is an integral part of Hadoop and effective job scheduling has a direct impact on Hadoop performance. Data locality is one of the most important factors to be considered in order to improve efficiency, as it affects data transmission through the system. A number of researchers have suggested approaches for improving data locality, but few have considered cache locality. In this paper, we present a state-of-the-art job scheduler, CLQLMRS (Cache Locality with Q-Learning in MapReduce Scheduler) for improving both data locality and cache locality using reinforcement learning. The proposed algorithm is evaluated by various experiments in a heterogeneous environment. Experimental results show significantly decreased execution time compared with FIFO, Delay, and the Adaptive Cache Local scheduler. [ABSTRACT FROM AUTHOR]

We consider an extended form of the MX/M/c queue with two types of server groups: Static as well as dynamic (which turn on/off in a state-dependent manner) servers. The two server groups may have homogenous or non-homogenous service rates. The model is further extended to feature setup and delayed-off times, finite capacity, and k staffing levels. This class of queues is solved via the difference equations approach, which addresses narratives in the literature and achieves higher numerical efficiency than the direct method. While the model of this queueing system is not new, the methodology for solving it is. Comparisons between our model and classic queues are provided followed by concluding remarks, including a summary of key observations. [ABSTRACT FROM AUTHOR]

Job scheduling in MapReduce plays a vital role in Hadoop performance. In recent years, many researchers have presented job scheduler algorithms to improve Hadoop performance. Designing a job scheduler that minimizes job execution time with maximum resource utilization is not a straightforward task. The primary purpose of this paper is to investigate agents affecting job scheduler efficiency and present a novel classification for job schedulers based on these factors. We provide a comprehensive overview of existing job schedulers in each group, evaluating their approaches, their effects on Hadoop performance, and comparing their advantages and disadvantages. Finally, we provide recommendations on choosing a preferred job scheduler in different environments for improving Hadoop performance. [ABSTRACT FROM AUTHOR]

Let HT ht , HT ht , HT ht be the cost rate per station per unit time for the server being busy, warming and idling respectively (we assume that the cost rate of a server that is switched off is zero). When a server idles, a stand-by period is initiated, where if no arrival is routed to that server, the server turns off. Jobs arrive to the system according to a Poisson process with rate HT ht . [Extracted from the article]

This paper presents a system for jointly managing cooling units and workload assignment in modular data centers. The system aims to minimize power consumption while respecting temperature constraints, all in a thermally heterogeneous environment. Unlike traditional cooling controllers, which may over/under cool certain areas in the data center due to the use of a single setpoint, our framework does not have a single setpoint to satisfy. Instead, using a data-driven thermal model, the proposed system generates an optimal temperature map, the required temperature distribution matrix (RTDM), to be used by the controller, eliminating under/over cooling and improving power efficiency. The RTDM is the resulting temperature distribution when jointly considering workload assignment and cooling control. In addition, we propose the use of model predictive control (MPC) to regulate the operational variables of cooling units in a power-efficient fashion to comply with the RTDM. Within each iteration of the MPC loop, an optimization problem involving the thermal model is solved, and the underlying thermal model is updated. To prove the feasibility of the proposed power efficient system, it has been implemented on an actual modular data center in our facilities. Results from the implementation show the potential for considerable power savings compared to other control methods. [ABSTRACT FROM AUTHOR]

This paper presents a three-stage algorithm for resource-aware scheduling of computational jobs in a large-scale heterogeneous data center. The algorithm aims to allocate job classes to machine configurations to attain an efficient mapping between job resource request profiles and machine resource capacity profiles. The first stage uses a queueing model that treats the system in an aggregated manner with pooled machines and jobs represented as a fluid flow. The latter two stages use combinatorial optimization techniques to solve a shorter-term, more accurate representation of the problem using the first-stage, long-term solution for heuristic guidance. In the second stage, jobs and machines are discretized. A linear programming model is used to obtain a solution to the discrete problem that maximizes the system capacity given a restriction on the job class and machine configuration pairings based on the solution of the first stage. The final stage is a scheduling policy that uses the solution from the second stage to guide the dispatching of arriving jobs to machines. We present experimental results of our algorithm on both Google workload trace data and generated data and show that it outperforms existing schedulers. These results illustrate the importance of considering heterogeneity of both job and machine configuration profiles in making effective scheduling decisions. [ABSTRACT FROM AUTHOR]