The Rising Tide of Liquid Cooling Solutions for Datacentres

Traditional air conditioners use a refrigerant gas and this type of closed-loop air-based cooling system is perfectly adequate for most for server rooms and small datacentres. However, as server rack power densities continue to increase due to machine learning applications, larger datacentres and hyperscale operators are turning to liquid cooling.

Most datacentre operators don’t want to disclose too much about their critical infrastructure arrangements as they are a source of their competitive advantage. However, some hyperscale cloud operators, have formed a working group for an open specification for liquid-cooled racks. These operators include Alphabet’s Google, Baidu, Facebook and Microsoft and one of the main uses for liquid cooling in their facilities will be for server racks housing IT systems and processors involved in machine learning.

As Hyperscale technologies start to move from Cloud applications to the network Edge, applications for liquid cooling solutions will continue to rise with the principle drivers being:

• Server processors and rack densities
• Storage system cooling needs
• Cooling Edge computer networks
• Crypto currency mining and IoT applications
• Power recovery and energy efficiency

Server Processors and Rack Densities

The amount of cooling required within a server room or datacentre application is directly proportional to the number of server processing units and their type. Whilst CPU (Central Processing Unit) performance growth is slowing (based on Moore’s Law), newer processing technologies are rapidly being deployed.

Graphic Processor Units (GPUs) are more powerful and are designed to accelerate graphics rendering and GPU-based servers are the most commonly used hardware accelerator at this moment in time. TPUs (Tensor Processing Units) are AI (Artificial Intelligence) Application Specific Integrated Circuits (ASICs) used by a hyperscale operator like Google, in its deep-learning project. Google’s TPUs are a direct-to-chip liquid cooling design.

A GPU will typically need about 200W of cooling and this can rise to 1kW or more of cooling requirement when the GPU is coupled to a high-performance CPU. In a typical application this will see single server rack power draws of 10-30kW or higher. As more GPUs are added inside such a rack and power densities increase, liquid cooling becomes a more viable option with modular cooling options from 20kW plus and upwards.

Storage System Cooling Needs

It is not just servers that require cooling. Storage solutions and their power densities are also evolving at a rapid rate.

Datacentres have traditionally used non-sealed hard drives that cannot be cooled except with air-based systems. Solid State Drives (SSDs) could offer an alternative solution and one that could be used with liquid or even complete immersion cooling solutions. Helium atmosphere Hard Disk Drives (HDDs) must be sealed by design and are another application for liquid-based cooling.

Whilst most server rooms operate a mixture of local and Cloud based storage, liquid cooling for storage systems is a viable solution for Cloud and hyperscale datacentres.

Cooling Edge Computer Networks

Edge computing is starting to become a mainstream consideration for service providers. Edge technology provides a way for datacentre operators to personalise and reduce the latency of centralised data operations by pushing these closer to the user at network edge.

A typical application could be a retail store or industrial factory connected to the Internet of Things (IoT). More of the applications run within edge-based locations will use high-performance computing and high-density storage systems which will need an efficient and reliable cooling technology. Liquid cooling provides an alternative where traditional cooling solutions cannot be installed or where the existing system cannot cater for the greater loads placed upon it. A typical rack window for liquid cooling would be 40-50kW per rack.

Crypto Currency Mining and IoT Applications

Financial institutions have always driven datacentre operations and pushed their designs for speed and power density. This trend has increased with more datacentres set-up or taken-over specifically for new technologies including Blockchain management and Bitcoin-type cryptocurrency mining.

Outside of the financial industry, other verticals that will form part of the Internet of Things (IoT) will require high-density and more powerful computing. A typical liquid cooling solution could run from 80-100kW per rack and full immersion tank cooling could be considered as a viable option such applications.

Power Recovery and PUE Ratio

One aspect that can be overlooked, is that more efficient cooling can recover power capacity. Liquid cooling solutions can be retrofitted into an existing datacentre or server room environment where an existing air-based cooling solution cannot be upgraded to cope with increased cooling demands.

Cooling is a must-have critical infrastructure system, and it is one that is also expensive to run. Liquid cooling and even immersion cooling solutions can be installed as a day-one solution or retrofitted to an existing server room or datacentre. Increased costs over traditional air-based cooling systems can be recovered in time through improvements in energy efficiency, and lower Power Usage Effectiveness (PUE) ratio. Energy efficiency improvements of over 90% are achievable when waste heat is redistributed for other areas of the datacentre or surrounding area.

Switching over to a liquid-based system should also provide the server room or datacentre site with a cooling solution better able to meet the rising power demands of processors and accelerators.

Quite rightly, water leakage in a datacentre environment will always be a concern. A traditional solution would be to install water leakage monitoring but this would only alarm when there was an actual leak and risk of damage to critical systems. Water leakage should not be an issue for a correctly installed and maintained liquid-based cooling system and one installed with its own leak prevention system (LPS).

Please contact us if you would like to discuss a cooling project or to arrange and air conditioning site survey for your server room or datacentre application.