Search

Growing demand for dense and high-performing IT compute capacity to support deep learning and artificial intelligence workloads necessitates data centers to look for more robust thermal management strategies. Today, data centers across the world are turning to liquid-based cooling solutions to keep up with the increased cooling demand for high power racks approaching 100kW of heat dissipation. Deploying direct-to-chip cold plate liquid cooling is one of the mainstream approaches which allows targeted cooling of high-power processors. This study provides the framework for a hybrid in row cooler (IRC) with liquid-to-air (L2A) heat exchanger (HX) system delivering chilled coolant to liquid-cooling cold plates mounted to the high heat dissipation electronics. This approach is useful for high heat density cooling of racks where no primary facility coolant is available at the data center. The present study aims to investigate the thermo-hydraulic performance of a distinct L2A IRC system that supplies cold secondary coolant (PG 25%) into the cooling loops of liquid-cooled servers in racks within an existing air-cooled data center. Thermal test vehicles (TTVs) are built to replicate actual high heat density servers. From the cold plate to data center level the proper choice of each level component was described based on their cooling performance and relevance. Three different cooling loop/rack designs are characterized experimentally, and detailed analytical and numerical (FNM) simulations are developed to analyze the heat exchanger performance. The FNM and CFD model of a data center are done in two steady and transient forms to study the performance of the L2A IRC in a data center.

Removal of heat is becoming a major challenge in today’s data centers. Computing-intensive applications such as artificial intelligence and machine learning are pushing data center to compute intensive systems, such as GPU, CPU, and switches to their extreme limits. Racks of IT can approach up to 100kW of heat dissipation challenging traditional data center designs for enterprises and cloud service providers. Direct-to-chip liquid cooling utilizing cold plates is becoming a common method of removing heat from high heat density data center server racks. There are various methods of applying liquid cooling to data centers to address the high heat density components such as liquid to liquid (L2L), liquid to air (L2A), and liquid to single phase refrigerant (L2R). This study aims to investigate the thermo-hydraulic performance of the L2L cooling systems using cooling distribution units (CDUs). CDUs provide a cold secondary coolant (Propylene Glycol 25%) into the cooling loops of liquid-cooled server racks, with the CDUs providing liquid to liquid heat exchange between the primary facility water and secondary liquid used for cold plates. This study uses Thermal Test Vehicles (TTVs) which have been built to reproduce and simulate high heat density servers. Four different cooling loops are characterized experimentally, and detailed analytical and numerical simulations using CFD are developed for analyzing the cooling characteristics of the entire L2L cooling loop, including the CDU, for removing heat from the cold plates. Detailed Flow Network Modeling (FNM) has been performed to analyze precise hydraulic modeling of the secondary fluid flow, from the CDUs to the cooling loops, for predicting pressure drop and flow rate of the secondary coolant. A FNM properly sizes the pumping requirements of the L2L cooling system. Additionally, a system calculator has been created for quickly sizing all secondary loop piping for L2L heat exchanger deployments.

Cold plates are at the heart of pumped liquid cooling systems. In this paper, we report on combined experimental, analytical, and computational efforts to characterize and model the thermal performance of advanced cold plates in order to establish their performance limits. A novel effectiveness-NTU formulation is introduced that models the fin array as a secondary “pseudo-fluid” such that accurate crossflow effectiveness models can be utilized to model the cold plates using well-known formulations. Experimental measurements and conjugate CFD simulations were made on cold plates with fin and channel features of order 100 um with water-propylene glycol (PG) mixtures as coolants. We show that for a fixed fin geometry, the best thermal performance, regardless of the pressure drop, is achieved when the flow rate is high enough to approach the low NTU convective limit which occurs for NTU approaching zero. For the model cold plate evaluated in this study, the lowest thermal resistance achieved at a flow rate of 4 LPM was 0.01 C/W, and the convective limit was 0.005 C/W. However, for a fixed pressure drop, the optimal cold plate should be designed to meet its TDP at the highest possible effectiveness in which the lower limit of thermal resistance is the advective limit achieved for NTU > 7. For the tested cold plate the advective limit for the thermal resistance is 0.003 C/W, but this limit can only be achieved if it is practically feasible to increase the surface area and heat transfer coefficient to maximize NTU for a targeted TDP.

The rising demand for high-performance central and graphical processing units has resulted in the need for more efficient thermal management techniques like direct-to-chip liquid cooling. Direct Liquid Cooling using cold plates is one of the most efficient and investigated cooling technologies since the 1980s. Major data and cloud providers are actively deploying liquid-cooled data center infrastructure due to rising computational demands. Liquid to liquid heat exchangers used in liquid-cooled data centers is also referred to as coolant distribution units (CDUs). Most of these CDUs selected by the data center operator is based on the heat load of the data center and the available head with that CDU. In this study, three 52U racks with six high-power TTV-based servers (Thermal Test Vehicles) in each rack were designed and deployed. Each server consists of eight GPU TTVs and six NV switch heaters. A 450-kW liquid-cooled CDU is used, and propylene glycol 25% is used as a coolant. Typical CDUs are designed to operate at 20 to 30% of the rated heat load to achieve a stable secondary coolant supply temperature. The present study will investigate the operations of CDU at very low heat loads, like 1% to 10% of the CDU’s rated capacity. At these low loads, large fluctuations in secondary side supply temperature were observed. This large fluctuation can lead to the failure of the 3-way valve used in CDUs at the primary side. In this paper, a control strategy is developed to stabilize the secondary supply temperature within ± 0.5 °C at very low loads using the combination of a flow control valve on the primary side and PID control settings within the CDU.

Increasing demands for cloud-based computing and storage, Internet-of-Things, and machine learning-based applications have necessitated the utilization of more efficient cooling technologies. Direct-to-chip liquid cooling using cold plates has proven to be one of the most efficient methods to dissipate the high heat fluxes of modern high-power CPUs and GPUs. While the published literature has well-documented research on the thermal aspects of direct liquid cooling, a detailed account of transient hydraulic investigation is still missing. In this experiment, a total of four 52U racks with four high-power TTV-servers (Thermal Test Vehicles) in each rack were designed and deployed. Each server consists of eight GPU TTVs and six NV switch heaters. Each of the two racks has a different vendor rack manifold and cooling loop modules (CLM). A 450 kW coolant distribution unit (CDU) is used to supply 25% propylene glycol coolant to these racks. Each rack has its own rack-level flow control valve to maintain the same flow rate. The present study provides an in-depth analysis of hydraulic transients when rack-level flow control valves are used with and without flow control. The operating conditions of the CDU are varied for different parameters, such as a constant flow rate, constant differential pressure, and constant pump speed. Furthermore, hydraulic transient is examined when the cooling loop modules are decommissioned from the rack one by one. The effect of this step-by-step decommissioning is assessed on the CDU operation and other racks. The pressure drop-based control strategy has been developed to maintain the same flow rate in the remaining servers in the rack when some cooling loop modules are decommissioned.

In a data center, electronic equipment such as server and switches dissipate heat and the corresponding cooling systems contribute to typically 25–35% of total energy consumption. The heat load continues to increase as there is a greater need for miniaturization and convergence. In 2014, data centers in the U.S. consumed an estimated 70 billion kWh, representing about 1.8% of total U.S. electricity consumption. Based on current trend estimates, U.S. data centers are projected to consume approximately 73 billion kWh in 2020 [1]. Many research and strategies are adopted to minimize energy cost. The recommended dry bulb temperature for long-term operation and reliability for air cooling is between 18–27°C and the largest allowable inlet temperature range to operate at is between 5°C and 45°C with American Society of Heating, Refrigeration, and Air-Conditioning Engineers (ASHRAE) enabling much broader allowable zones) [2]. But understanding a proper cooling system is very important especially for thermal management of IT equipment with high heat loads such as 1U or 2U multi-core, high-end servers and blade servers which provide more computing per watt. Many problems like high inlet temperature due to the mixing of hot air with cold air, local hot spots, lower system reliability, increased failure, and downtime may occur. Among many other approaches to managing high-density racks, in-row coolers are used in between racks to provide cold air and minimize local hot spots. This paper describes a computational study being performed by applying in-row coolers for different rack power configuration with and without aisle containment. The power, as well as the number of racks, are varied to study the effect of raised inlet temperature for the IT equipment in a Computational Fluid Dynamics (CFD) model developed in 6SigmaRoom with the help of built-in library items. A comparative analysis is also performed for a typical small-sized non-raised facility to investigate the efficacy and limitations of in-row coolers in thermal management of IT equipment with variation in rack heat load and containment. Several other aspects like a parametric study of variable opening areas of duct between racks and in-row coolers, the variation of operating flow rate and failure scenarios are also studied to find proper flow distribution, uniformity of outlet temperature and predict better performance, energy savings and reliability. The results are presented for general guidance for flexible and quick installation and safe operation of in-row coolers to improve thermal efficiency.

Complete immersion of servers in synthetic dielectric fluids is rapidly becoming a popular technique to minimize the energy consumed by data centers for cooling purposes. In general, immersion cooling offers noteworthy advantages over conventional air-cooling methods as synthetic dielectric fluids have high heat dissipation capacities which are roughly about 1200 times greater than air. Other advantages of dielectric fluid immersion cooling include even thermal profile on chips, reduction in noise and addressing reliability and operational enhancements like whisker formation and electrochemical migration. Nevertheless, lack of data published and availability of long-term reliability data on immersion cooling is insufficient which makes most of data centers operators reluctant to implement this technique. The first part of this paper will compare thermal performance of single-phase oil immersion cooled HP ProLiant DL160 G6 server against air cooled server using computational fluid dynamics on 6SigmaET®. Focus of the study are major components of the server like Central Processing Unit (CPU), Dual in Line Memory Module (DIMM), Input/output Hub (IOH) chip and Input/output controller Hub (ICH). The second part of this paper focuses on thermal performance optimization of oil immersion cooled servers by varying inlet oil temperature, flow rate and using different fluid.

In typical data centers, the servers and IT equipment are cooled by air and almost half of total IT power is dedicated to cooling. Hybrid cooling is a combined cooling technology with both air and water, where the main heat generating components are cooled by water or water-based coolants and rest of the components are cooled by air supplied by CRAC or CRAH. Retrofitting the air-cooled servers with cold plates and pumps has the advantage over thermal management of CPUs and other high heat generating components. In a typical 1U server, the CPUs were retrofitted with cold plates and the server tested with raised coolant inlet conditions. The study showed the server can operate with maximum utilization for CPUs, DIMMs, and PCH for inlet coolant temperature from 25–45 °C following the ASHRAE guidelines. The server was also tested for failure scenarios of the pumps and fans with reducing numbers of fans and pumps. To reduce cooling power consumption at the facility level and increase air-side economizer hours, the hybrid cooled server can be operated at raised inlet air temperatures. The trade-off in energy savings at the facility level due to raising the inlet air temperatures versus the possible increase in server fan power and component temperatures is investigated. A detailed CFD analysis with a minimum number of server fans can provide a way to find an operating range of inlet air temperature for a hybrid cooled server. Changes in the model are carried out in 6SigmaET for an individual server and compared to the experimental data to validate the model. The results from this study can be helpful in determining the room level operating set points for data centers housing hybrid cooled server racks.

Modern day data centers are operated at high power for increased power density, maintenance, and cooling which covers almost 2 percent (70 billion kilowatt-hours) of the total energy consumption in the US. IT components and cooling system occupy the major portion of this energy consumption. Although data centers are designed to perform efficiently, cooling the high-density components is still a challenge. So, alternative methods to improve the cooling efficiency has become the drive to reduce the cooling cost. As liquid cooling is more efficient for high specific heat capacity, density, and thermal conductivity, hybrid cooling can offer the advantage of liquid cooling of high heat generating components in the traditional air-cooled servers. In this experiment, a 1U server is equipped with cold plate to cool the CPUs while the rest of the components are cooled by fans. In this study, predictive fan and pump failure analysis are performed which also helps to explore the options for redundancy and to reduce the cooling cost by improving cooling efficiency. Redundancy requires the knowledge of planned and unplanned system failures. As the main heat generating components are cooled by liquid, warm water cooling can be employed to observe the effects of raised inlet conditions in a hybrid cooled server with failure scenarios. The ASHRAE guidance class W4 for liquid cooling is chosen for our experiment to operate in a range from 25°C – 45°C. The experiments are conducted separately for the pump and fan failure scenarios. Computational load of idle, 10%, 30%, 50%, 70% and 98% are applied while powering only one pump and the miniature dry cooler fans are controlled externally to maintain constant inlet temperature of the coolant. As the rest of components such as DIMMs & PCH are cooled by air, maximum utilization for memory is applied while reducing the number fans in each case for fan failure scenario. The components temperatures and power consumption are recorded in each case for performance analysis.

The energy consumption in a typical data center is rising day by day to meet the increasing demand of high compute servers resulting in high energy costs. In typical data centers, almost half of the IT energy is dedicated to cooling of the IT server racks. Hybrid cooling technology as an approach to minimize the power consumption uses of water or water based fluid to cool a high heat generating component whereas the rest of the components is cooled by air using internal fans. In this paper, our objective is to optimize the flow rate of air cooling loop of such hybrid cooled server to make data center cooling more cost effective and energy efficient. The volume of air supplied is controlled by varying the air flow rate through the internal fans. Also, the number of fans is reduced from five to three to minimize the power consumption. This finding helped us realize the 40% decrease in Fan power for each server. The pressure difference between the cold and hot aisle is created to avoid the backflow of cold air. Parameters like CPU and memory utilization were varied and temperature of different components such as processors, Dual-In-Line-Memory Module (DIMMs) & Platform Controller Hub (PCH) was monitored. When the server is running with 3 fans (approx. 10 CFM) maximum temperatures are found below the critical temperature. Further studies are also carried out through CFD analysis to observe changes in a model in 6SigmaET to increase the cooling efficiency. The study gives an idea of power savings by fan consolidation for energy efficient hybrid servers.

In response to the growing demand for power and energy across US and world-wide, development of energy efficient solutions has become very important. Considering data center applications, cooling power consumption constitutes a significant part of the overall energy usage of the system. In the process of optimizing the energy consumed per performance unit, liquid cooling has become one of the key solutions. Liquid cooling addresses the critical issues related to typical air cooling in servers because of its better heat transfer characteristics. Water-cooling at the device level can be an efficient solution since water has higher thermal capacitance when compared to traditional heat carrying medium i.e., air. Additionally, moving the cooling source closure to the heat load helps in reducing hot spots at the room level, thereby, improving the effectiveness of the cooling solution. On the other hand, the feasibility to operate liquid cooling at wider temperature range helps in partially or eliminating the room air conditioning units and chiller compressors and reduces the cooling power consumptions substantially. The primary objective of the study is to evaluate air vs. liquid cooled enterprise-class server for the effects of high ambient inlet temperatures. The air-cooled server has five hot swappable fans and two CPU heat sinks to dissipate the total heat in the server. The water-cooled counter-part consists of two-cold plates with integrated pumps to cool the CPUs while the dual in-line memory modules transfer the heat using a heat transfer tape attached to the manifold that is carrying the coolant. The existing practice in the industry is to maximize the use of economizer usage by reducing/eliminating the usage of chiller while taking advantage of outside ambient conditions to cool the data centers. However, higher ambient temperatures would lead to higher operating temperatures of the processors. The operating temperature of the CPU has direct influence on the static power which is known to reduce the performance of the processor. In this study, theair as well as liquid cooled servers will be tested at inlet temperatures ranging from 25°C-45°C (ASHRAE T.C 9.9 A4 and W4 classes) and the corresponding thermal and flow parameters will be compared for both cases. The current work primarily focuses on evaluating the critical inlet temperatures (for air cooled vs. liquid cooled server) at which CPU die temperatures has an influence of the static power. Collectively, a case will be presented as to why l iquid cooling would provide a viable option to operate the data centers at wider inlet temperatures with reduced risk of leakage current and sustain the performance and reliability of the IT equipment.