Search

Data centres increase their size and complexity due to the increasing amount of heterogeneous work loads and patterns to be served. Such a mix of various purpose workloads makes the optimisation of resource management systems according to temporal or application-level patterns difficult. Data centre operators have developed multiple resource-management models to improve scheduling perfor mance in controlled scenarios. However, the constant evolution of the workloads makes the utilisation of only one resource-management model sub-optimal in some scenarios. In this work, we propose: (a) a machine learning regression model based on gradient boosting to pre dict the time a resource manager needs to schedule incoming jobs for a given period; and (b) a resource management model, Boost, that takes advantage of this regression model to predict the scheduling time of a catalogue of resource managers so that the most performant can be used for a time span. The benefits of the proposed resource-management model are analysed by comparing its scheduling performance KPIs to those provided by the two most popular resource-management models: two level, used by Apache Mesos, and shared-state, employed by Google Borg. Such gains are empirically eval uated by simulating a hyper-scale data centre that executes a realistic synthetically generated workload that follows real-world trace patterns

Data centres constitute the engine of the Internet, and run a major portion of large web and mobile applications, content delivery and sharing platforms, and Cloud-computing business models. The high performance of such infrastructures is therefore critical for their correct functioning. This work focuses on the improvement of data-centre performance by dynamically switching the main data-centre governance software system: the resource manager. Instead of focusing on the development of new resource-managing models as soon as new workloads and patterns appear, we propose DISCERNER, a decision-theory model that can learn from numerous data-centre execution logs to determine which existing resource-managing model may optimise the overall performance for a given time period. Such a decision-theory system employs a classic machine-learning classifier to make real-time decisions based on past execution logs and on the current data-centre operational situation. A set of extensive and industry-guided experiments has been simulated by a validated data-centre simulation tool. The results obtained show that the values of key performance indicators may be improved by at least 20% in realistic scenarios.

Carbon emissions, greenhouse gases and pollution in general are usually related to traditional factories, so the most modern computing factories have gone unnoticed for the general-public opinion. We empirically show through extensive and realistic simulation that: 1) energy consumption, and consequently CO2 emissions, could be reduced from ~15% to ~60% if the correct energy-efficiency policies are applied; and 2) such energy-consumption reduction can be achieved without negatively impacting the correct operation of these infrastructures. To this end, this work is focused on the proposal and analysis of a set of energy-efficiency policies which are applied to traditional and hyper-scale data centres, as well as numerous operation environments, including: 1) the top resource managers used in industry; 2) eight energy-efficiency policies, including aggressive, fine-tuned and adaptive models; and 3) three types of workload-arrival patterns. Finally, we present a realistic analysis of the environmental impact of the application of such energy-efficiency policies on USA data centres. The presented results estimate that 11.5 million of tons of CO2 could be saved, which is equivalent to the removal of 4.79 million of combustion cars, that is, the total car fleet of countries such as Portugal, Austria and Sweden.

Data centres have become the backbone of large Cloud services and applica-tions, providing virtually unlimited elastic and scalable computational and storage resources. The search for the efficiency and optimisation of resources is one of the current key aspects for large Cloud Service Providers and is becoming more and more challenging, since new computing paradigms such as Internet of Things, Cyber-Physical Systems and Edge Computing are spreading. One of the key aspects to achieve efficiency in data centres consists of the discovery and proper analysis of the data-centre behaviour. In this paper, we present a model to automatically retrieve execution workflows of existing data-centre logs by employing process mining tech-niques. The discovered processes are characterised and analysed according to the understandability and complexity in terms of execution efficiency of data-centre jobs. We finally validate and demonstrate the usability of the proposal by applying the model in a real scenario, that is, the Google Cluster traces

Edge computing constitutes a key paradigm to address the new requirements of areas such as smart cars, industry 4.0, and health care, where massive amounts of heterogeneous data from continuous geographically-distributed sources have to be processed and computed near real-time. To this end, new distributed infrastructures consisting on small computing clusters close to data sources, also known as Cloudlets have emerged. In order to evaluate the performance of these solutions we present Sphere, a simulation tool that enables researchers to establish various scenarios, including: (a) topology and orchestration model of the infrastructure; (b) incoming workload patterns; (c) resource-managing models; and (d) scheduling policies. Moreover, Sphere allows researchers to apply efficiency and performance policies both at infrastructure and cluster levels. The simulator presents the following benefits: (a) Evaluation of various orchestration models; (b) Analysis of resource-efficiency and performance strategies at Edge-infrastructure and cluster (Cloudlet/Cloud) level; (c) Execution of diverse workload generation patterns; (d) Evaluation of strategies for the infrastructure communication, as well as the impact on tasks completion time (makespan); and (e) Simulation of each cluster (Cloudlet/Cloud) independently, including resource-managing, scheduling and resource-efficiency models. Finally, we performed a deep evaluation based on realistic Edge-Computing use cases. The results of the experiments confirm that it is a performant and reliable tool for the analysis of orchestration, graph-resolving, energy-efficiency, resource-managing and scheduling strategies in Edge-computing environments.

Data centres are quickly evolving to support new demands for Cloud-Computing services. Extreme workload peaks represent a challenge for the maintenance of the performance and service level agreements, even more when operation costs need to be minimised. In this paper, we first present an extensive analysis of the impact of extreme workloads in large-scale realistic Cloud-Computing data centres, including a comparison between the most relevant centralised resource managing models. Moreover, we extend our previous works by proposing a new energy-efficiency policy called Bullfighter which is able to keep performance key performance indicators while reducing energy consumption in extreme scenarios. This policy employs queue-theory distributions to foresee workload demands and adapt automatically to workload fluc tuations even in extreme environments, while avoiding the fine-tuning required for other energy policies. Finally, it is shown through extensive simulation that Bullfighter can save more than 40% of energy in the aforementioned scenarios without exerting any noticeable impact on data-centre performance.

The number of connected sensors and devices is expected to increase to billions in the near future. However, centralised cloud-computing data centres present various challenges to meet the requirements inherent to Internet of Things (IoT) workloads, such as low latency, high throughput and bandwidth constraints. Edge computing is becoming the standard computing paradigm for latency-sensitive real-time IoT workloads, since it addresses the aforementioned limitations related to centralised cloud-computing models. Such a paradigm relies on bringing computation close to the source of data, which presents serious operational challenges for large-scale cloud-computing providers. In this work, we present an architecture composed of low-cost Single-Board-Computer clusters near to data sources, and centralised cloud-computing data centres. The proposed cost-efficient model may be employed as an alternative to fog computing to meet real-time IoT workload requirements while keeping scalability. We include an extensive empirical analysis to assess the suitability of single-board-computer clusters as cost-effective edge-computing micro data centres. Additionally, we compare the proposed architecture with traditional cloudlet and cloud architectures, and evaluate them through extensive simulation. We finally show that acquisition costs can be drastically reduced while keeping performance levels in data-intensive IoT use cases.

This chapter presents the authors’ work for the Case Study entitled “Delivering Social Media with Scalability” within the framework of High-Performance Modelling and Simulation for Big Data Applications (cHiPSet) COST Action 1406. We identify some core research areas and give an outline of the publications we came up within the framework of the aforementioned action. The ease of user content generation within social media platforms, e.g. check-in information, multimedia data, etc., along with the proliferation of Global Positioning System (GPS)-enabled, always-connected capture devices lead to data streams of unprecedented amount and a radical change in information sharing. Social data streams raise a variety of practical challenges: derivation of real-time meaningful insights from effectively gathered social information, a paradigm shift for content distribution with the leverage of contextual data associated with user preferences, geographical characteristics and devices in general, etc. In this article we present the methodology we followed, the results of our work and the outline of a comprehensive survey, that depicts the state-of-the-art situation and organizes challenges concerning social media streams and the infrastructure of the data centers supporting the efficient access to data streams in terms of content distribution, data diffusion, data replication, energy efficiency and network infrastructure. The challenges of enabling better provisioning of social media data have been identified and they were based on the context of users accessing these resources. The existing literature has been systematized and the main research points and industrial efforts in the area were identified and analyzed. In our works, in the framework of the Action, we came up with potential solutions addressing the problems of the area and described how these fit in the general ecosystem.

The management and processing of the data generated by healthcare systems is getting more and more attention because of the digitalization of a traditionally analogical sector such as healthcare. Various data management frameworks and solutions have emerged to face some of the multiple challenges present in this environment, includ ing: (a) Data privacy and security; (b) Inter-operability between het erogeneous systems; and (c) Usability and readability of health-related sensitive information. In this paper, the authors share their experience on lecturing on how to address such issues by developing web-based software from several points of views: (a) Healthcare professional needs and requirements in terms of usability, accessibility and easiness; (b) Technical requirements and knowledge required to develop all the layers of a fully functional example of a micro health information system; and (c) Technical require ments to share and distribute the data required by several agents of the healthcare environment, focusing on the adoption of international stan dards such as HL7; (d) Perform of all operations in a secure, available and reliable way. It is concluded that the application of Flipped Teaching among with Project-based Learning may have a very positive impact on both in grades and drop rates for late-years (junior and senior) courses within the Health Engineering bachelor’s degrees.

Social media have emerged in the last decade as a viable and ubiquitous means of communication. The ease of user content generation within these platforms, e.g. check-in information, multimedia data, etc., along with the proliferation of Global Positioning System (GPS)-enabled, always-connected capture devices lead to data streams of unprecedented amount and a radical change in information sharing. Social data streams raise a variety of practical challenges, including derivation of real-time meaningful insights from effectively gathered social information, as well as a paradigm shift for content distribution with the leverage of contextual data associated with user preferences, geographical characteristics and devices in general. In this article we present a comprehensive survey that outlines the state-of-the-art situation and organizes challenges concerning social media streams and the infrastructure of the data centres supporting the efficient access to data streams in terms of content distribution, data diffusion, data replication, energy efficiency and network infrastructure. We systematize the existing literature and proceed to identify and analyse the main research points and industrial efforts in the area as far as modelling, simulation and performance evaluation are concerned.

Energy-awareness remains one of the main concerns for today's cloud computing (CC) operators. The optimisation of energy consumption in both cloud computational clusters and computing servers is usually related to scheduling problems. The definition of an optimal scheduling policy which does not negatively impact to system performance and task completion time is still challenging. In this work, we present a new simulation tool for cloud computing, GAME-SCORE, which implements a scheduling model based on the Stackelberg game. This game presents two main players: a) the scheduler and b) the energy-efficiency agent. We used the GAME-SCORE simulator to analyse the efficiency of the proposed game-based scheduling model. The obtained results show that the Stackelberg cloud scheduler performs better than static energy-optimisation strategies and can achieve a fair balance between low energy consumption and short makespan in a very short time

Cloud computing (CC) systems are the most popular computational environments for providing elastic and scalable services on a massive scale. The nature of such systems often results in energy-related problems that have to be solved for sustainability, cost reduction, and environment protection. In this paper we defined and developed a set of performance and energy-aware strategies for resource allocation, task scheduling, and for the hibernation of virtual machines. The idea behind this model is to combine energy and performance-aware scheduling policies in order to hibernate those virtual machines that operate in idle state. The efficiency achieved by applying the proposed models has been tested using a realistic large-scale CC system simulator. Obtained results show that a balance between low energy consumption and short makespan can be achieved. Several security constraints may be considered in this model. Each security constraint is characterized by: (a) Security Demands (SD) of tasks; and (b) Trust Levels (TL) provided by virtual machines. SD and TL are computed during the scheduling process in order to provide proper security services. Experimental results show that the proposed solution reduces up to 45% of the energy consumption of the CC system. Such significant improvement was achieved by the combination of an energy-aware scheduler with energy-efficiency policies focused on the hibernation of VMs.

Achieving efficiency both in terms of resource utilisation and energy consumption is acomplex challenge, especially in large-scale wide-purpose data centers that serve cloud- computing services. Simulation presents an appropriate solution for the development andtesting of strategies that aim to improve efficiency problems before their applications inproduction environments. Various cloud simulators have been proposed to cover differentaspects of the operation environment of cloud-computing systems. In this paper, we define the SCORE tool, which is dedicated to the simulation of energy-efficient monolithicand parallel-scheduling models and for the execution of heterogeneous, realistic and synthetic workloads. The simulator has been evaluated through empirical tests. The results ofthe experiments confirm that SCORE is a performant and reliable tool for testing energy- efficiency, security, and scheduling strategies in cloud-computing environments.

This Doctoral Dissertation, presented as a set of research contributions, focuses on resource efficiency in data centers. This topic has been faced mainly by the development of several energy-efficiency, resource managing and scheduling policies, as well as the simulation tools required to test them in realistic cloud computing environments. Several models have been implemented in order to minimize energy consumption in Cloud Computing environments. Among them: a) Fifteen probabilistic and deterministic energy-policies which shut-down idle machines; b) Five energy-aware scheduling algorithms, including several genetic algorithm models; c) A Stackelberg game-based strategy which models the concurrency between opposite requirements of Cloud-Computing systems in order to dynamically apply the most optimal scheduling algorithms and energy-efficiency policies depending on the environment; and d) A productive analysis on the resource efficiency of several realistic cloud–computing environments. A novel simulation tool called SCORE, able to simulate several data-center sizes, machine heterogeneity, security levels, workload composition and patterns, scheduling strategies and energy-efficiency strategies, was developed in order to test these strategies in large-scale cloud-computing clusters. As results, more than fifty Key Performance Indicators (KPI) show that more than 20% of energy consumption can be reduced in realistic high-utilization environments when proper policies are employed., Esta Tesis Doctoral, que se presenta como compendio de artículos de investigación, se centra en la eficiencia en la utilización de los recursos en centros de datos de internet. Este problema ha sido abordado esencialmente desarrollando diferentes estrategias de eficiencia energética, gestión y distribución de recursos, así como todas las herramientas de simulación y análisis necesarias para su validación en entornos realistas de Cloud Computing. Numerosas estrategias han sido desarrolladas para minimizar el consumo energético en entornos de Cloud Computing. Entre ellos: 1. Quince políticas de eficiencia energética, tanto probabilísticas como deterministas, que apagan máquinas en estado de espera siempre que sea posible; 2. Cinco algoritmos de distribución de tareas que tienen en cuenta el consumo energético, incluyendo varios modelos de algoritmos genéticos; 3. Una estrategia basada en la teoría de juegos de Stackelberg que modela la competición entre diferentes partes de los centros de datos que tienen objetivos encontrados. Este modelo aplica dinámicamente las estrategias de distribución de tareas y las políticas de eficiencia energética dependiendo de las características del entorno; y 4. Un análisis productivo sobre la eficiencia en la utilización de recursos en numerosos escenarios de Cloud Computing. Una nueva herramienta de simulación llamada SCORE se ha desarrollado para analizar las estrategias antes mencionadas en clústers de Cloud Computing de grandes dimensiones. Los resultados obtenidos muestran que se puede conseguir un ahorro de energía superior al 20% en entornos realistas de alta utilización si se emplean las estrategias de eficiencia energética adecuadas. SCORE es open source y puede simular diferentes centros de datos con, entre otros muchos, los siguientes parámetros: Tamaño del centro de datos; heterogeneidad de los servidores; tipo, composición y patrones de carga de trabajo, estrategias de distribución de tareas y políticas de eficiencia energ

Information technologies must be made aware of the sustainability of cost reduction. Data centers may reach energy consumption levels comparable to many industrial facilities and small-sized towns. Therefore, innovative and transparent energy policies should be applied to improve energy consumption and deliver the best performance. This paper compares, analyzes and evaluates various energy efficiency policies, which shut down underutilized machines, on an extensive set of data-center environments. Data envelopment analysis (DEA) is then conducted for the detection of the best energy efficiency policy and data-center characterization for each case. This analysis evaluates energy consumption and performance indicators for natural DEA and constant returns to scale (CRS). We identify the best energy policies and scheduling strategies for high and low data-center demands and for medium-sized and large data-centers; moreover, this work enables data-center managers to detect inefficiencies and to implement further corrective actions.

Cloud computing and data centers that support this paradigm are rapidly evolving in order to satisfy new demands. These ever-growing needs represent an energy-related challenge to achieve sustainability and cost reduction. In this paper, we define an expert and intelligent system that applies various en ergy policies. These policies are employed to maximize the energy-efficiency of data-center resources by simulating a realistic environment and heterogeneous workload in a trustworthy tool. An environmental and economic impact of around 20% of energy consumption can be saved in high-utilization scenarios without exerting any noticeable impact on data-center performance if an adequate policy is applied.

The quality of services in Cloud Computing (CC) depends on the scheduling strategies selected for processing of the complex workloads in the physical cloud clusters. Using the scheduler of the single type does not guarantee of the optimal mapping of jobs onto cloud resources, especially in the case of the processing of the big data workloads. In this paper, we compare the performances of the cloud schedulers for various combinations of the cloud workloads with different characteristics. We define several scenarios where the proper types of schedulers can be selected from a list of scheduling models implemented in the system, and used to schedule the concrete workloads based on the workloads’ parameters and the feedback on the efficiency of the schedulers. The presented work is the first step in the development and implementation of an automatic intelligent scheduler selection system. In our simple experimental analysis, we confirm the usefulness of such a system in today’s data-intensive cloud computing

Energy-awareness remians the important problem in today’s cloud computing (CC). Optimization of the energy consumed in cloud data centers and computing servers is usually related to the scheduling prob lems. It is very difficult to define an optimal schedul ing policy without negoative influence into the system performance and task completion time. In this work, we define a general cloud scheduling model based on a Stackelberg game with the workload scheduler and energy-efficiency agent as the main players. In this game, the aim of the scheduler is the minimization of the makespan of the workload, which is achieved by the employ of a genetic scheduling algorithm that maps the workload tasks into the computational nodes. The energy-efficiency agent selects the energy-optimization techniques based on the idea of switchin-off of the idle machines, in response to the scheduler decisions. The efficiency of the proposed model has been tested using a SCORE cloud simmulator. Obtained results show that the proposed model performs better than static energy-optimization strategies, achieving a fair balance between low energy consumption and short queue times and makespan.

This Doctoral Dissertation, presented as a set of research contributions, focuses on resource efficiency in data centers. This topic has been faced mainly by the development of several energy-efficiency, resource managing and scheduling policies, as well as the simulation tools required to test them in realistic cloud computing environments. Several models have been implemented in order to minimize energy consumption in Cloud Computing environments. Among them: a) Fifteen probabilistic and deterministic energy-policies which shut-down idle machines; b) Five energy-aware scheduling algorithms, including several genetic algorithm models; c) A Stackelberg game-based strategy which models the concurrency between opposite requirements of Cloud-Computing systems in order to dynamically apply the most optimal scheduling algorithms and energy-efficiency policies depending on the environment; and d) A productive analysis on the resource efficiency of several realistic cloud–computing environments. A novel simulation tool called SCORE, able to simulate several data-center sizes, machine heterogeneity, security levels, workload composition and patterns, scheduling strategies and energy-efficiency strategies, was developed in order to test these strategies in large-scale cloud-computing clusters. As results, more than fifty Key Performance Indicators (KPI) show that more than 20% of energy consumption can be reduced in realistic high-utilization environments when proper policies are employed., Esta Tesis Doctoral, que se presenta como compendio de artículos de investigación, se centra en la eficiencia en la utilización de los recursos en centros de datos de internet. Este problema ha sido abordado esencialmente desarrollando diferentes estrategias de eficiencia energética, gestión y distribución de recursos, así como todas las herramientas de simulación y análisis necesarias para su validación en entornos realistas de Cloud Computing. Numerosas estrategias han sido desarrolladas para minimizar el consumo energético en entornos de Cloud Computing. Entre ellos: 1. Quince políticas de eficiencia energética, tanto probabilísticas como deterministas, que apagan máquinas en estado de espera siempre que sea posible; 2. Cinco algoritmos de distribución de tareas que tienen en cuenta el consumo energético, incluyendo varios modelos de algoritmos genéticos; 3. Una estrategia basada en la teoría de juegos de Stackelberg que modela la competición entre diferentes partes de los centros de datos que tienen objetivos encontrados. Este modelo aplica dinámicamente las estrategias de distribución de tareas y las políticas de eficiencia energética dependiendo de las características del entorno; y 4. Un análisis productivo sobre la eficiencia en la utilización de recursos en numerosos escenarios de Cloud Computing. Una nueva herramienta de simulación llamada SCORE se ha desarrollado para analizar las estrategias antes mencionadas en clústers de Cloud Computing de grandes dimensiones. Los resultados obtenidos muestran que se puede conseguir un ahorro de energía superior al 20% en entornos realistas de alta utilización si se emplean las estrategias de eficiencia energética adecuadas. SCORE es open source y puede simular diferentes centros de datos con, entre otros muchos, los siguientes parámetros: Tamaño del centro de datos; heterogeneidad de los servidores; tipo, composición y patrones de carga de trabajo, estrategias de distribución de tareas y políticas de eficiencia energ

The fear experienced by data center administrators presents an ongoing problem due to the low percentage of machines that they are willing to switch off in order to save energy. This risk aversion can be assessed from acognitive system. The purpose of this paper is to demonstrate the extra costs in curred by maintaining all the machines of a data center executing continuously for fear of damaging hardware, degradating the service, or losing data. To this end, an objective function which minimizes energy consumption depending on the number of times that the machines ares witched on/off is provided. The risk version experienced by these data center administrator scan be measured from the percentage of machines that they are willing to switch off. It is shown that it is always the best option to turn off machines in order to reduce costs, given a formulation of the cognitive aspects of the fear experienced by data center administrators.

Activity recognition (AR) and fall detection (FD) research areas are very related in assistance scenarios but evolve independently. Evaluate them is not trivial and the lack of FD real-world datasets implies a big issue. A protocol that fuses AR and FD is proposed to achieve a large, open and growing dataset that could, potentially, provide an enhanced understanding of the activities and fall process and the information needed to design and evaluate high-performance systems.

With the increase of intelligent devices, ubiquitous computing is spreading to all scopes of people life. Smart home (or industrial) environments include automation and control devices to save energy, perform tasks, assist and give comfort in order to satisfy specific preferences. This paper focuses on the proposal for Software Reference Architecture for the development of smart applications and their deployment in smart environments. The motivation for this Reference Architecture and its benefits are also explained. The proposal considers three main processes in the software architecture of these applications: perception, reasoning and acting. This paper centres attention on the definition of the Perception process and provides an example for its implementation and subsequent validation of the proposal. The software presented implements the Perception process of a smart environment for a standard office, by retrieving data from the real world and storing it for further reasoning and acting processes. The objectives of this solution include the provision of comfort for the users and the saving of energy in lighting. Through this verification, it is also shown that developments under this proposal produce major benefits within the software life cycle.

This work studies the problem of energy consumption growth in two spheres: Grid-Computing and Smart Environments. These problems are tackled through the establishment of energy-saving policies developed for each environment in order to save the maximum energy as possible. In the Grid-Computing environment, seven energypolicies were designed in an attempt to minimize energy consumption through shutting resources down and booting them. It is proved that approximately 40% of energy can be saved. Efficiency of various grid locations was compared using Data Envelopment Analysis (DEA) methodology. In Smart Environments where sensors perceive lighting conditions, the energy-saving policy adjusts lighting in order to satisfy user preferences and prevents energy from being wasted. A set of wireless sensors were deployed on two offices at the department of Computer Languages and Systems. The dataset created over several months was employed to extract information about user lighting preferences, from the application of which it is proven that around 70% of energy can be saved in lighting appliances.

This work studies the problem of energy consumption growth in two spheres: Grid-Computing and Smart Environments. These problems are tackled through the establishment of energy-saving policies developed for each environment in order to save the maximum energy as possible. In the Grid-Computing environment, seven energypolicies were designed in an attempt to minimize energy consumption through shutting resources down and booting them. It is proved that approximately 40% of energy can be saved. Efficiency of various grid locations was compared using Data Envelopment Analysis methodology. In Smart Environments where sensors perceive lighting conditions, the energy-saving policy adjusts lighting in order to satisfy user preferences and prevents energy from being wasted. A set of wireless sensors were deployed on two offices at the department of Computer Languages and Systems. The dataset created over several months was employed to extract information about user lighting preferences, from the application of which it is proven that around 70% of energy can be saved in lighting appliances.

The interest about Smart City concept has in creased in recent years. In fact, Smart Cities is ex pected to improve cityzens life experience by driv ing the next digital revolution, moving from the personal area (mobile computing, smart home) to the urban area (collective computing and collective intelligence). But the development of Smart Cities is not being as fast as expected. Several problems need to be undertaken in order to achieve the ob jectives of the paradigm. This paper presents an approach to address one of these problems: to or chestrate the platform that is required for gathering information about city, store it in a model and ena ble it for exploitation. The heterogeneity of the po tential data sources available and the complexity of the information nature and structure, make it a non trivial task that have to be solved before commer cial solutions appear and provide specific and non interoperable solutions.