Search

Cooling failure in data centers (DCs) is a complex phenomenon due to the many interactions between the cooling infrastructure and the information technology equipment (IT). To fully understand it, a system integration philosophy is vital to the testing and design of experiment. In this paper, a facility-level DC cooling failure experiment is run and analyzed. An airside cooling failure is introduced to the facility during two different cooling set points as well as in open and contained environments. Quantitative instrumentation includes pressure differentials, tile airflow, external contour and discrete air inlet temperature, intelligent platform management interface (IPMI), and cooling system data during failure recovery. Qualitative measurements include infrared imaging and airflow visualization via smoke trace. To our knowledge of current literature, this is the first experimental study in which an actual multi-aisle facility cooling failure is run with real IT (compute, network, and storage) load in the white space. This will establish a link between variations from the facility to the central processing unit (CPU). The results show that using the external IT inlet temperature sensors, the containment configuration shows a longer available uptime (AU) during failure. However, the IPMI data show the opposite. In fact, the available uptime is reduced significantly when the external sensors are compared to internal IT analytics. The response of the IT power, CPU temperature, and fan speed shows higher values during the containment failure. This occurs because of the instantaneous formation of external impedances in the containment during failure, which renders the contained aisle to be less resilient than the open aisle. The tradeoffs between PUE, OPEX, and AU are also explained.

The rapid growth in the number of data centers combined with the high-density heat dissipation of computer and telecommunications equipment has made energy efficient thermal management of data centers a key research area. Localized hybrid air–water cooling is one approach to more effectively control the cooling when there is wide variation in the amount of dissipation in neighboring racks while the traditional air cooling approach requires overprovisioning. In a closed, hybrid air–water cooled server cabinet, the generated heat is removed by a self-contained system that does not interact with the room level air cooling system. Here, a hybrid-cooled enclosed cabinet and all its internal components were characterized experimentally in steady-state mode (e.g., experimentally determined heat-exchanger effectiveness and IT characterization). Also, a comprehensive numerical model of the cabinet was developed and validated using the experimental data. The computational model employs full numerical modeling of the cabinet geometry and compact models to represent the servers and the air/water heat exchanger. The compact models were developed based on experimental flow and thermal characterization of the internal components. The cabinet level model has been used to simulate a number of operating scenarios relevant to data center applications such as the effect of air leakage within the cabinet. The effect of the air side and the water side failure of the cooling system on the IT performance were investigated experimentally. A comparison was made of the amount of time required to exceed the operating temperature limit for the two scenarios.

Today's data centers increasingly rely on environmental data collection and analysis to operate the cooling infrastructure as efficiently as possible and to maintain the reliability of IT equipment. This in turn emphasizes the importance of the quality of data collected and its relevance to the overall operation of the data center. This study presents an experiment based analysis and comparison of environmental and power data collection using two different approaches; one uses a discrete sensor network and smart PDUs, and another uses available data from the installed IT equipment (IPMI data). The comparison looks deeply into the effect of both approaches that are adopted to control data center cooling. In addition, the effect that the Supply Air Temperature (SAT) from the Computer Room Air Handler (CRAH) unit had on the IT equipment was investigated in fully sealed Cold Aisle Containment (CAC) with 100% CPU utilization. It can be observed that the difference between the discrete and IPMI inlet temperature of the IT equipment increased as SAT increased due to the IT fans increasing speed in an attempt to get more cooling and the resulting in negative pressure differential build up inside the containment. Furthermore, the authors identified a value of the supply air temperature at which IT equipment started to ramp up for both approaches of data center cooling and control. The novelty of this study may aid data center operators when making the decision of what monitoring or control scheme to use.

In this investigation, we utilize a new method of characterization in which IT airflow systems are ranked from a 1RU ToR switches, xRU servers to 9RU Blade Center. The free delivery (FD) and the critical pressure (Pc) are unique properties of each IT airflow system, satisfying its active flow curve (AFC). Those curves are used in previously validated models to investigate the interaction of IT with different air systems in a confined space. A validated model of containment is studied of a CAC aisle from Binghamton University data center lab. It is shown that reverse flow can take place in an operating piece of IT with weaker air system; this can endanger IT reliability and up time at different data center events. This can also change previous conclusions about the ride through time in well-sealed fully contained systems. Another important finding is that design containment systems based on the assumption of uniform load in the aisle/cabinet is a misconception that can endanger facility operation. In general, the reduction in airflow rate upon events in the data center (DC) is a function of each IT air system. To our knowledge, this is the first time in literature where both recent models of IT active flow curves and black box thermal inertia are combined to yield realistic failure analysis of contained solutions.

Data Centers are prone to power outages and cooling failures. During such events, complex transport interactions take place between the cooling system and the IT. Empirical data on this phenomenon is scarce in the current literature due to the complexity and size of such experiments. In this study, a facility level data center blowers cooling failure experiment is run and analyzed. Quantitative instrumentation includes pressure differentials, tile airflow, point air inlet temperature, contours air inlet temperature and IT IPMI data during failure-recovery. Qualitative measurements include IR imaging and airflow visualization via smoke trace. To our knowledge, this is the first experimental study in literature in which an actual multi aisle facility cooling failure is run with real IT (compute, Network and storage) load in the white space. This will enable a link between variations from the facility to the chip levels. Results show that by using external air inlet temperature sensors the containment configuration has a longer uptime during failure. However, the IPMI data shows the opposite. In fact, the RTT is reduced by ∼70% when the external and internal sensors are compared. This occurs due external impedances formed by the containment during failure degrading IT airflow systems. The inconsistency between IT IPMI inlet sensors and externally placed IT or rack inlet sensors (based on best practices) are expected to increase as the airflow imbalances increase.

In cold aisle containment (CAC) the supply of cold air is separated within the contained volume. The hot air exhaust leaves the IT and increases the room’s temperature before returning to the cooling unit. On the other hand, hot aisle containment (HAC) generates a cooler environment in the data center room as a whole by segregating hot air within the containment. Hot air is routed back to the cooling unit return by a drop ceiling or a chimney. Each system has different characteristics and airflow paths. For instance, leakage introduces different effects for CACs and HACs since the hot and cold aisles are switched. This article utilizes data center measurements and containment characterization carried out circa April 2015 in the ES2 Data Center lab at Binghamton University. Details on the containment model include leakages at below racks, above racks, below CAC doors, between doors, and above doors. The model deploys the experimentally obtained flow curves approach for flow-pressure correlation. Data center operators rely on the pressure differential to measure how much the IT is provided. Hence, in this study the level of provisioning was expressed in terms of pressure differentials between the hot and cold aisles. In this manner the model reflected real-life DC thermal management practices. This was done by integrating a pressure differential based controller to the cooling unit model. Leakages in each system are quantified and ranked based on a proposed LIF (Leakage Impact Factor) metric. The LIF describes the transport contribution each leakage location has. This metric can be used by containment designers and data center operators to prioritize their sealing efforts or consider deploying the containment solution differently. Finally, a systematic approach is shown in which containment models can be used to optimize operations at the real-life site.

Cooling power constitutes a large portion of the total electrical power consumption in data centers. Approximately 25%∼40% of the electricity used within a production data center is consumed by the cooling system. Improving the cooling energy efficiency has attracted a great deal of research attention. Many strategies have been proposed for cutting the data center energy costs. One of the effective strategies for increasing the cooling efficiency is using dynamic thermal management. Another effective strategy is placing cooling devices (heat exchangers) closer to the source of heat. This is the basic design principle of many hybrid cooling systems and liquid cooling systems for data centers. Dynamic thermal management of data centers is a huge challenge, due to the fact that data centers are operated under complex dynamic conditions, even during normal operating conditions. In addition, hybrid cooling systems for data centers introduce additional localized cooling devices, such as in row cooling units and overhead coolers, which significantly increase the complexity of dynamic thermal management. Therefore, it is of paramount importance to characterize the dynamic responses of data centers under variations from different cooling units, such as cooling air flow rate variations. In this study, a detailed computational analysis of an in row cooler based hybrid cooled data center is conducted using a commercially available computational fluid dynamics (CFD) code. A representative CFD model for a raised floor data center with cold aisle-hot aisle arrangement fashion is developed. The hybrid cooling system is designed using perimeter CRAH units and localized in row cooling units. The CRAH unit supplies centralized cooling air to the under floor plenum, and the cooling air enters the cold aisle through perforated tiles. The in row cooling unit is located on the raised floor between the server racks. It supplies the cooling air directly to the cold aisle, and intakes hot air from the back of the racks (hot aisle). Therefore, two different cooling air sources are supplied to the cold aisle, but the ways they are delivered to the cold aisle are different. Several modeling cases are designed to study the transient effects of variations in the flow rates of the two cooling air sources. The server power and the cooling air flow variation combination scenarios are also modeled and studied. The detailed impacts of each modeling case on the rack inlet air temperature and cold aisle air flow distribution are studied. The results presented in this work provide an understanding of the effects of air flow variations on the thermal performance of data centers. The results and corresponding analysis is used for improving the running efficiency of this type of raised floor hybrid data centers using CRAH and IRC units.

The perpetual increase of data processing has led to an ever increasing need for power and in turn to greater cooling challenges. High density (HD) IT loads have necessitated more aggressive and direct approaches of cooling as opposed to the legacy approach by the utilization of row-based cooling. In-row cooler systems are placed between the racks aligned with row orientation; they offer cool air to the IT equipment more directly and effectively. Following a horizontal airflow pattern and typically occupying 50% of a rack’s width; in-row cooling can be the main source of cooling in the data center or can work jointly with perimeter cooling. Another important development is the use of containment systems since they reduce mixing of hot and cold air in the facility. Both in-row technology and containment can be combined to form a very effective cooling solution for HD data centers. This current study numerically investigates the behavior of in-row coolers in cold aisle containment (CAC) vs. perimeter cooling scheme. Also, we address the steady state performance for both systems, this includes manufacturer’s specifications such as heat exchanger performance and cooling coil capacity. A brief failure scenario is then run, and duration of ride through time in the case of row-based cooling system failure is compared to raised floor perimeter cooling with containment. Non-raised floor cooling schemes will reduce the air volumetric storage of the whole facility (in this small data center cell it is about a 20% reduction). Also, the varying thermal inertia between the typical in-row and perimeter cooling units is of decisive importance. The CFD model is validated using a new data center laboratory at Binghamton University with perimeter cooling. This data center consists of one main Liebert cooling unit, 46 perforated tiles with 22% open area, 40 racks distributed on three main cold aisles C and D. A computational slice is taken of the data center to generalize results. Cold aisle C consists of 16 rack and 18 perforated tiles with containment installed. In-row coolers are then added to the CFD model. Fixed IT load is maintained throughout the simulation and steady state comparisons are built between the legacy and row-based cooling schemes. An empirically obtained flow curve method is used to capture the flow-pressure correlation for flow devices. Performance scenarios were parametrically analyzed for the following cases: (a) Perimeter cooling in CAC, (b) In-row cooling in CAC. Results showed that in-row coolers increased the efficiency of supply air flow utilization since the floor leakage was eliminated, and higher pressure build up in CAC were observed. This reduced the rack recirculation when compared to the perimeter cooled case. However, the heat exchanger size demonstrated the limitation of the in-row to maintain controlled set point at increased air flow conditions. For the pump failure scenario, experimental data provided by Emerson labs were used to capture the thermal inertia effect of the cooling coils for in-row and perimeter unit, perimeter cooled system proved to have longer ride through time.

The heat dissipated by electronic equipment inside data centers is increasing at a rapid rate due to the increasing of performance requirement and package density. This ever increasing power leads to critical challenges of thermal management for these high power density data centers. Energy consumption is also a key issue for high density data centers. Roughly 1.5% of all U.S. electricity consumption in the year 2006 was related to data centers, while that number increased to 2% by the year 2010. In 2013, U.S. data centers consumed approximately 91 billion kilowatt-hours of electricity. This amount of the electricity equals the annual output of 34 500-megawatt coal-fired power plants [1]. Cooling systems constitute a significant portion of the energy consumption of data centers, being approximately 25%∼35% of the total energy usage. Therefore, there is a large potential to save energy by optimizing current existing cooling systems and investigating new cooling technologies, and, at the same time, improving the overall cooling capacity and efficiency. This paper describes and investigates a hybrid cooling technology which utilizes in row coolers in existing raised floor air cooled data centers. The in row cooler functions as a liquid-to-air heat exchanger. In addition to the traditional raised floor cold aisle-hot aisle arrangements, the in row cooler is installed between the IT equipment to enable delivering the liquid coolant medium closer to the IT equipment. The in row coolers intake the hot air from the hot aisle, condition it, and supply the chilled air to the cold aisle. Thus, by extracting a large portion of the heat more directly into the cooling liquid through the in row coolers compared with the perimeter CRAH unit, the overall cooling performance and efficiency can potentially be improved. CFD models for an in row cooler and a representative data center room are developed. Experimentally characterized performance data are used to calibrate and validate the models. The models are then used to conduct a detailed computational analysis to assess the effectiveness of different arrangement configurations of in row cooler units in two rows of racks along one cold aisle. The detailed performance of the entire cold aisle is characterized using the rack inlet air temperature and a temperature nonuniformity factor. The impact of CRAH location and room layout are also investigated. This study is based on a practical problem and the corresponding results and analysis provide basic installation and design guidelines for future equipment upgrading in certain parts of the data center.Copyright © 2015 by ASME

The heat dissipated by high performance IT equipment such as servers and switches in data centers is increasing rapidly, which makes the thermal management even more challenging. IT equipment is typically designed to operate at a rack inlet air temperature ranging between 10 °C and 35 °C. The newest published environmental standards for operating IT equipment proposed by ASHARE specify a long term recommended dry bulb IT air inlet temperature range as 18°C to 27°C. In terms of the short term specification, the largest allowable inlet temperature range to operate at is between 5°C and 45°C. Failure in maintaining these specifications will lead to significantly detrimental impacts to the performance and reliability of these electronic devices. Thus, understanding the cooling system is of paramount importance for the design and operation of data centers. In this paper, a hybrid cooling system is numerically modeled and investigated. The numerical modeling is conducted using a commercial computational fluid dynamics (CFD) code. The hybrid cooling strategy is specified by mounting the in row cooling units between the server racks to assist the raised floor air cooling. The effect of several input variables, including rack heat load and heat density, rack air flow rate, in row cooling unit operating cooling fluid flow rate and temperature, in row coil effectiveness, centralized cooling unit supply air flow rate, non-uniformity in rack heat load, and raised floor height are studied parametrically. Their detailed effects on the rack inlet air temperatures and the in row cooler performance are presented. The modeling results and corresponding analyses are used to develop general installation and operation guidance for the in row cooler strategy of a data center.Copyright © 2015 by ASME