Search

Growing demand for dense and high-performing IT compute capacity to support deep learning and artificial intelligence workloads necessitates data centers to look for more robust thermal management strategies. Today, data centers across the world are turning to liquid-based cooling solutions to keep up with the increased cooling demand for high power racks approaching 100kW of heat dissipation. Deploying direct-to-chip cold plate liquid cooling is one of the mainstream approaches which allows targeted cooling of high-power processors. This study provides the framework for a hybrid in row cooler (IRC) with liquid-to-air (L2A) heat exchanger (HX) system delivering chilled coolant to liquid-cooling cold plates mounted to the high heat dissipation electronics. This approach is useful for high heat density cooling of racks where no primary facility coolant is available at the data center. The present study aims to investigate the thermo-hydraulic performance of a distinct L2A IRC system that supplies cold secondary coolant (PG 25%) into the cooling loops of liquid-cooled servers in racks within an existing air-cooled data center. Thermal test vehicles (TTVs) are built to replicate actual high heat density servers. From the cold plate to data center level the proper choice of each level component was described based on their cooling performance and relevance. Three different cooling loop/rack designs are characterized experimentally, and detailed analytical and numerical (FNM) simulations are developed to analyze the heat exchanger performance. The FNM and CFD model of a data center are done in two steady and transient forms to study the performance of the L2A IRC in a data center.

Due to increased use of high-performance computing in datacenters to cater to huge workloads, old low-performance compute servers must be replaced endlessly with high-performance compute servers. Traditional air-cooling systems are insufficient to provision and run the servers in optimal conditions as the datacenter thermal footprint or rack density grows, resulting in thermal throttling. To sustain the growing needs, Rear Door Heat Exchangers (RDHx) are deployed in existing datacenters along with peripheral Computer Room Air Handling/Conditioning (CRAH/CRAC) units. RDHx transfers heat from the rear end of the racks and rejects it into the facility’s chilled water. This study will demonstrate the suitability of RDHx for low density as well as high density rack applications. A baseline CFD model had a generic datacenter layout with peripheral CRAH/CRAC units and RDHx. Several case studies were conducted by varying the air and liquid inlet temperatures for rack and RDHx, respectively. We also compared active and passive modes of operating RDHx while server fans provide flowrate based on the IT inlet temperature. The paper will also discuss the feasibility of designing a datacenter with only RDHx and no peripheral CRAC/CRAH units while maintaining the thermal envelop. The research will also provide a guideline in implementing RDHx based on the heat load and server design.

In recent years, there has been a significant increase in cloud computing, networking, virtualization, and storage applications, leading to an increase in demand for high-performance servers. The increase in performance demands is currently being met by increasing CPU and GPU power densities that require more efficient cooling technologies as compared to air traditional cooling methods. Cold plate-based liquid cooling in air-cooled servers enables efficient thermal management with minimal changes to existing air-cooling infrastructure. In a hybrid cooled server, the demand for air cooling is reduced as the primary heat-generating components are indirectly cooled by cold plates. In this study, experiments are performed with optimized chassis of a hybrid cooled Cisco C220 server. The chassis design is optimized to improve the airflow by providing additional vents on the chassis to allow more low-temperature airflow rather than the heated airflow approaching from the drive bay. Also, the design of the heat sink baffle is improved which allows a more streamlined flow to approach the heat sinks. This is done by designing and manufacturing a new 3-D printed baffle. This optimized baffle design helps in reducing the pressure drop across the system hence helping in the reduction of fan speeds and reducing the fan power consumption. Results are generated by iterating the fan speed and inlet temperature of air and comparing them with the baseline design of the server. Conclusions are made on the reduction in fan power due to the improved chassis design and any reduction in temperatures of air-cooled components.

Removal of heat is becoming a major challenge in today’s data centers. Computing-intensive applications such as artificial intelligence and machine learning are pushing data center to compute intensive systems, such as GPU, CPU, and switches to their extreme limits. Racks of IT can approach up to 100kW of heat dissipation challenging traditional data center designs for enterprises and cloud service providers. Direct-to-chip liquid cooling utilizing cold plates is becoming a common method of removing heat from high heat density data center server racks. There are various methods of applying liquid cooling to data centers to address the high heat density components such as liquid to liquid (L2L), liquid to air (L2A), and liquid to single phase refrigerant (L2R). This study aims to investigate the thermo-hydraulic performance of the L2L cooling systems using cooling distribution units (CDUs). CDUs provide a cold secondary coolant (Propylene Glycol 25%) into the cooling loops of liquid-cooled server racks, with the CDUs providing liquid to liquid heat exchange between the primary facility water and secondary liquid used for cold plates. This study uses Thermal Test Vehicles (TTVs) which have been built to reproduce and simulate high heat density servers. Four different cooling loops are characterized experimentally, and detailed analytical and numerical simulations using CFD are developed for analyzing the cooling characteristics of the entire L2L cooling loop, including the CDU, for removing heat from the cold plates. Detailed Flow Network Modeling (FNM) has been performed to analyze precise hydraulic modeling of the secondary fluid flow, from the CDUs to the cooling loops, for predicting pressure drop and flow rate of the secondary coolant. A FNM properly sizes the pumping requirements of the L2L cooling system. Additionally, a system calculator has been created for quickly sizing all secondary loop piping for L2L heat exchanger deployments.

Rising power densities at the server level due to increasing performance demands are being met by using efficient thermal management methods such as direct-to-chip liquid cooling. The use of cold plates that are directly installed yields a lower thermal resistance path from the chip to the ambient. In a hybrid-cooled server arrangement, high-heat-generating components are cooled with water or a water-based fluid, while the rest of the components are cooled with air using server-level fans. It is imperative to characterize the heat capture ratio for various server boundary conditions to ascertain the best possible liquid and airflow rates and temperatures. These parameters serve as inputs in defining the Total Cost of Ownership (TCO). The present investigation numerically evaluates the heat capture ratio in a hybrid cooled server for peak server load and varying inlet temperature for air and liquid. The CFD model of a Cisco Series C220 server with direct-to-chip liquid-cooled CPUs was developed. The cold plate for the CPU was experimentally characterized for pressure drop and thermal resistance characteristics and a black-box model was used for CFD simulations using 25% propylene glycol as the coolant. The heat capture ratio value was obtained under the varied temperature and flow rate boundary conditions of air and liquid. Based on the heat capture ratio values obtained, optimum values of inlet temperatures and flow rates are recommended for air and liquid for the server being investigated.

The rising demand for high-performance central and graphical processing units has resulted in the need for more efficient thermal management techniques like direct-to-chip liquid cooling. Direct Liquid Cooling using cold plates is one of the most efficient and investigated cooling technologies since the 1980s. Major data and cloud providers are actively deploying liquid-cooled data center infrastructure due to rising computational demands. Liquid to liquid heat exchangers used in liquid-cooled data centers is also referred to as coolant distribution units (CDUs). Most of these CDUs selected by the data center operator is based on the heat load of the data center and the available head with that CDU. In this study, three 52U racks with six high-power TTV-based servers (Thermal Test Vehicles) in each rack were designed and deployed. Each server consists of eight GPU TTVs and six NV switch heaters. A 450-kW liquid-cooled CDU is used, and propylene glycol 25% is used as a coolant. Typical CDUs are designed to operate at 20 to 30% of the rated heat load to achieve a stable secondary coolant supply temperature. The present study will investigate the operations of CDU at very low heat loads, like 1% to 10% of the CDU’s rated capacity. At these low loads, large fluctuations in secondary side supply temperature were observed. This large fluctuation can lead to the failure of the 3-way valve used in CDUs at the primary side. In this paper, a control strategy is developed to stabilize the secondary supply temperature within ± 0.5 °C at very low loads using the combination of a flow control valve on the primary side and PID control settings within the CDU.

Increasing demands for cloud-based computing and storage, Internet-of-Things, and machine learning-based applications have necessitated the utilization of more efficient cooling technologies. Direct-to-chip liquid cooling using cold plates has proven to be one of the most efficient methods to dissipate the high heat fluxes of modern high-power CPUs and GPUs. While the published literature has well-documented research on the thermal aspects of direct liquid cooling, a detailed account of transient hydraulic investigation is still missing. In this experiment, a total of four 52U racks with four high-power TTV-servers (Thermal Test Vehicles) in each rack were designed and deployed. Each server consists of eight GPU TTVs and six NV switch heaters. Each of the two racks has a different vendor rack manifold and cooling loop modules (CLM). A 450 kW coolant distribution unit (CDU) is used to supply 25% propylene glycol coolant to these racks. Each rack has its own rack-level flow control valve to maintain the same flow rate. The present study provides an in-depth analysis of hydraulic transients when rack-level flow control valves are used with and without flow control. The operating conditions of the CDU are varied for different parameters, such as a constant flow rate, constant differential pressure, and constant pump speed. Furthermore, hydraulic transient is examined when the cooling loop modules are decommissioned from the rack one by one. The effect of this step-by-step decommissioning is assessed on the CDU operation and other racks. The pressure drop-based control strategy has been developed to maintain the same flow rate in the remaining servers in the rack when some cooling loop modules are decommissioned.

Data center proliferation has been increasing significantly around the world attributed to growth in technologies such as the Internet of Things (IoT), bitcoin mining, and high-performance computing (HPC). A direct consequence of these developments is an enhancement in processing units and a corresponding rise in CPU and GPU power densities. Limitations of air cooling to dissipate increasing power densities in processors have compelled the researchers to move toward better and efficient liquid cooling solutions. In an earlier study, a custom-made mini-rack with liquid-cooled 2OU (open rack Unit) web servers were tested for comparison of centralized and distributed pumping with constant flow rates at the server level. The effect of higher inlet temperature in terms of IT power, cooling power consumption, and CPU temperature was reported along with a comparison of centralized versus distributed pumping. In this article, the same 2OU server is used to show the effect of variable flow rate on server thermal performance. The parameters monitored for performance quantification are the core temperatures, dual in-line memory modules (DIMM) temperatures, and platform controller hub (PCH) temperature. These parameters were reported by varying the CPU power consumption, coolant inlet temperatures, and coolant flow rates by controlling the distributed pumps. The server was fully enclosed with no outside air intake where the CPUs were cooled by cold plates and the rest of the components with internally recirculating air. The results obtained from the experiments were compared with the results obtained from a previous study where the effect of the variable flow rate was ignored. A full-factorial design of experiments (DoE) was designed using Minitab 19 to analyze the results statistically.

Continuous rise in cloud computing and other web-based services propelled the data center proliferation seen over the past decade. Traditional data centers use vapor-compression-based cooling units that not only reduce energy efficiency but also increase operational and initial investment costs due to involved redundancies. Free air cooling and airside economization can substantially reduce the information technology equipment (ITE) cooling power consumption, which accounts for approximately 40% of energy consumption for a typical air-cooled data center. However, this cooling approach entails an inherent risk of exposing the ITE to harmful ultrafine particulate contaminants, thus, potentially reducing the equipment and component reliability. The present investigation attempts to quantify the effects of particulate contamination inside the data center equipment and ITE room using computational fluid dynamics (CFD). An analysis of the boundary conditions to be used was done by detailed modeling of ITE and the data center white space. Both two-dimensional and three-dimensional simulations were done for detailed analysis of particle transport within the server enclosure. An analysis of the effect of the primary pressure loss obstructions like heat sinks and dual inline memory modules inside the server was done to visualize the localized particle concentrations within the server. A room-level simulation was then conducted to identify the most vulnerable locations of particle concentration within the data center space. The results show that parameters such as higher velocities, heat sink cutouts, and higher aspect ratio features within the server tend to increase the particle concentration inside the servers.

Transistor density trends till recently have been following Moore's law, doubling every generation resulting in increased power density. The computational performance gains with the breakdown of Moore's law were achieved by using multicore processors, leading to nonuniform power distribution and localized high temperatures making thermal management even more challenging. Cold plate-based liquid cooling has proven to be one of the most efficient technologies in overcoming these thermal management issues. Traditional liquid-cooled data center deployments provide a constant flow rate to servers irrespective of the workload, leading to excessive consumption of coolant pumping power. Therefore, a further enhancement in the efficiency of implementation of liquid cooling in data centers is possible. The present investigation proposes the implementation of dynamic cooling using an active flow control device to regulate the coolant flow rates at the server level. This device can aid in pumping power savings by controlling the flow rates based on server utilization. The flow control device design contains a V-cut ball valve connected to a microservo motor used for varying the device valve angle. The valve position was varied to change the flow rate through the valve by servomotor actuation based on predecided rotational angles. The device operation was characterized by quantifying the flow rates and pressure drop across the device by changing the valve position using both computational fluid dynamics and experiments. The proposed flow control device was able to vary the flow rate between 0.09 lpm and 4 lpm at different valve positions.

Over the last decade, several hyper-scale data center companies such as Google, Facebook, and Microsoft have demonstrated the cost-saving capabilities of airside economization with direct/indirect heat exchangers by moving to chiller-less air-cooled data centers. Under pressure from data center owners, information technology equipment OEMs like Dell and IBM are developing information technology equipment that can withstand peak excursion temperature ratings of up to 45 °C, clearly outside the recommended envelope, and into ASHRAEs A4 allowable envelope. As popular and widespread as these cooling technologies are becoming, airside economization comes with its challenges. There is a risk of premature hardware failures or reliability degradation posed by uncontrolled fine particulate and gaseous contaminants in presence of temperature and humidity transients. This paper presents an in-depth review of the particulate and gaseous contamination-related challenges faced by the modern-day data center facilities that use airside economization. This review summarizes specific experimental and computational studies to characterize the airborne contaminants and associated failure modes and mechanisms. In addition, standard lab-based and in-situ test methods for measuring the corrosive effects of the particles and the corrosive gases, as the means of testing the robustness of the equipment against these contaminants, under different temperature and relative humidity conditions are also reviewed. It also outlines the cost-sensitive mitigation techniques like improved filtration strategies and methods that can be utilized for efficient implementation of airside economization.

In today’s world, most data centers have multiple racks with numerous servers in each of them. The high amount of heat dissipation has become the largest server-level cooling problem for the data centers. The higher dissipation required, the higher is the total energy required to run the data center. Although still the most widely used cooling methodology, air cooling has reached its cooling capabilities especially for High-Performance Computing data centers. Liquid-cooled servers have several advantages over their air-cooled counterparts, primarily of which are high thermal mass, lower maintenance. Nano-fluids have been used in the past for improving the thermal efficiency of traditional dielectric coolants in the power electronics and automotive industry. Nanofluids have shown great promise in improving the convective heat transfer properties of the coolants due to a proven increase in thermal conductivity and specific heat capacity. The present research investigates the thermal enhancement of the performance of de-ionized water-based dielectric coolant with Copper nanoparticles for a higher heat transfer from the server cold plates. Detailed 3-D modeling of a commercial cold plate is completed and the CFD analysis is done in a commercially available CFD code ANSYS CFX. The obtained results compare the improvement in heat transfer due to improvement in coolant properties with data available in the literature.

With more development in electronics system capable of having larger functional densities, power density is increasing. Immersion cooling demonstrates the highest power usage efficiency (PUE) among all cooling techniques for data centers and there is still interest in optimizing immersion cooling to use it to its full potential. The aim of this paper is to present the effect of inclination and thermal shadowing on two-phase immersion cooling using FC-72. For simulation of boiling, the RPI (Rensselaer Polytechnic Institute) wall boiling model has been used. Also, two empirical models were used for calculation of bubble departure diameter and nucleate site density. The boundary condition was assumed to be constant heat flux and the bath temperature was kept at boiling temperature of FC-72 and the container pressure is assumed to be atmospheric. this study showed that due to the thermal shadowing, boiling boundary layer can lay over the top chipset and increases vapor volume fraction over top chipsets. This ultimately causes increase in maximum temperature of second chip. The other main observation is with higher inclination angle of chip, maximum temperature on the chip decreases up to 3°C.

Modern-day data center administrators are finding it increasingly difficult to lower the costs incurred in mechanical cooling of their IT equipment. This is especially true for high-performance computing facilities like Artificial Intelligence, Bitcoin Mining, and Deep Learning, etc. Airside Economization or free air cooling has been out there as a technology for a long time now to reduce the mechanical cooling costs. In free air cooling, under favorable ambient conditions of temperature and humidity, outside air can be used for cooling the IT equipment. In doing so, the IT equipment is exposed to sub-micron particulate/gaseous contaminants that might enter the data center facility with the cooling airflow. The present investigation uses a computational approach to model the airflow paths of particulate contaminants entering inside the IT equipment using a commercially available CFD code. A Discrete Phase Particle modeling approach is chosen to calculate trajectories of the dispersed contaminants. Standard RANS approach is used to model the airflow in the airflow and the particles are superimposed on the flow field by the CFD solver using Lagrangian particle tracking. The server geometry was modeled in 2-D with a combination of rectangular and cylindrical obstructions. This was done to comprehend the effect of change in the obstruction type and aspect ratio on particle distribution. Identifying such discrete areas of contaminant proliferation based on concentration fields due to changing geometries will help with the mitigation of particulate contamination related failures in data centers.

Modern data center facilities administrators are finding it increasingly difficult to lower the costs incurred in mechanical cooling of their IT equipment. This is especially true for high computing applications like Artificial Intelligence, Bitcoin Mining, Deep Learning, etc. Airside Economization/free air cooling reduces the mechanical cooling costs by using outside air to cool IT equipment under favorable ambient conditions. In this process, administrators risk their equipment to the exposure of fine particulate/ gaseous contaminants that might enter the data center facility with the cooling airflow. Literature suggests that the nature of failures caused by particulate contamination is very intermittent which makes the failures tough to predict. While the recommended filters can remove PM 10-2.5 , it's the fine and ultra-fine particulates like DPM (Diesel Particulate Matter), corrosive salts of high ionic content like sulfates and nitrates with low DRH (Deliquescent Relative Humidity) values that are the cause of concern. The present investigation utilizes a 3 - D CFD modeling of particle-laden flow in a rectangular flow domain, imitating the flow through floor tiles as in a raised floor data center. Literature was reviewed to study various numerical models that have been used for simulating particle dispersion and particle deposition in ventilated rooms, air ducts and particle behavior across physical obstructions of various geometries. A Discrete Phase Modeling approach was chosen using ANSYS FLUENT to calculate trajectories of the dispersed contaminants. 6SigmaRoom was used to predict accurate boundary and flow conditions of the fluid flow leaving the floor tiles.