How Rack Power Densities Influence Data Centre Design

The rack power density calculation is one of the most fundamental when it comes to server room and data centre designs. The calculation is based on a summation of the total kilowatts (kW) of power consumed by all the devices within each server cabinet. Multiplied by the total number of cabinets or server racks in the room, the total provides the basis for capacity planning, sizing critical power protection and cooling systems.

Comparing Server Room and Data Centre Racks

The Uptime Institute recently released their 2020 survey on rack power densities and this showed that densities of 20kW or more where becoming more common within data centre environments with most respondents reporting power densities within 10-19kW. Some data centres reported rack power demands as high as 30kW and above. When these are removed from the survey data, the average for 2020 is around 8.4kW/rack.

More info: https://journal.uptimeinstitute.com/rack-density-is-rising/

Reviewing our own UK server room projects from last year, we note a smaller average kw/rack figure of 6kW per rack. This lower figure highlights the difference in survey populations with the majority of our 2020 projects taking place in on-premises server rooms run by commercial businesses and organisations rather than enterprise or colocation-type data centres. UK on-premise Server rooms tend to operate a smaller number of physical servers and rack cabinets, typically up to 6 on average.

The Uptime Institute survey focuses on data centre environments and here high-end computing demand is driving up rack power densities. Examples include cryptocurrency mining, augmented and virtual reality, artificial intelligence, Edge computing and Internet of Things (IoT) applications are more computer intensive. Add to this list the roll out of 5G in the UK and a dramatic rise in remote home working, and it is not hard to imagine that the average rack power density could increase in 2021 and beyond in both larger data centres and smaller server rooms.

For server room and data centre design engineers the differences in average rack power densities open up different approaches to the design and installation of critical power and cooling infrastructure systems.

Uninterruptible Power Supplies

In previous years energy efficiency and modularity have been two of the main technology drivers within the uninterruptible power supplies (UPS) industry. The modular UPS concept was once an outlier with only a handful of companies offering this type of arrangement as a competitor to the more traditional monoblock type UPS system. Now almost every UPS manufacturer offers a modular system with some modules sized as low as 10kW (3phase) with the average being 25-50kW. There is now a 100kW module on the market from at least one manufacturer and it is forecast that more UPS manufacturers will launch this size of module within the next two years.

A key advantage of a modular UPS design is its scalability. The UPS configuration can be right-sized from day-one and adapted as the load increases (or decreases) through the addition (or removal) of hot-swap power modules. This also assists power consumption and energy efficiency for those interesting in reporting their power usage effectiveness (PUE) and reducing their operating costs. Modular UPS do by default tend to also be highly energy efficient with load profiles that can maintain upwards of 96-97% energy efficiency with load profiles as low as 25%. Monoblock UPS designs have also benefited from advancements in power electronics and can show similar energy efficiency levels but lack the ease with which to provide N+X redundancy.

For data centres, modular and monoblock designs lend themselves to centralised power protection plans. The UPS system is sized to power the critical and essential loads within the facility. There should also be an on-site generator to reduce the overall size of battery needed and ensure operational and business continuity for major power outages that could last up to several hours or even days.

Below 10kW, for three phase and single phase UPS applications, some UPS manufacturers have also launched new (or updated) monobloc UPS ranges specially for rack applications and to meet rising rack power densities. These systems tend be available ‘dual’ combination units that can be installed in either rack or floor standing tower formats. Highly efficiency (96% or greater), these types of UPS also offer scalability through plug-in parallel operation cards. A typical system also offers extended runtime battery packs.

These smaller single and three phase UPS systems enable a decentralised approach to critical power protection. One or more uninterruptible power supplies are installed within the server room. A UPS can be installed to protect the loads within the same rack it is installed into.

For server room applications, these types of UPS provide on-line power protection with extended runtime options that can be added to later. Providing there is space within the rack (or beside the rack), the additional UPS can be installed to provide N+X resilient or increased capacity or both. Whilst single phase UPS tend to be the norm for more rack installations, as power densities rise, three phase UPS (3/3 or 3/1 configurations) may be required to meet the needs of rising rack power densities.

Lead Acid and Lithium Batteries

The most used battery in an uninterruptible power supply application is one using lead-acid technology. Lead acid is particularly compatible with standby power applications where there is a need for intermittent backup power and long periods during which to recharge.

Lithium UPS batteries are a development from the battery technology used energy storage and electric vehicle (EV) applications. Lithium batteries are more suited to fast charge/discharge cycles. They are also less susceptible to heat damage and so may be more suitable for the higher ambient temperatures suggested for server rooms and data centres by ASHRAE (American Society of Heating, Refrigerating and Air-Conditioning Engineers).

It is not surprising that lithium UPS or UPS using lithium-ion batteries are now available for use within server rooms and data centres. Uninterruptible power supplies with lithium-ion (Li-ion) batteries are available as floor standing and 19inch rack mount devices. They are more compact than a traditional lead acid battery UPS but have a price premium.

There are two reasons for the higher costs. One is volume related as lead acid batteries are manufactured in far greater volumes worldwide compared to lithium. As appetite for electric vehicles increases and electricity grids start to deploy more domestic and grid-scale energy storage systems, production volumes will grow, and the cost of lithium batteries should start to fall. The second factor is the more sophisticated charging and management electronics required for lithium batteries. One of the main concerns with a lithium battery set is thermal runaway, which can in turn lead to heat and fire issues. Lithium itself catches fire when exposed to oxygen and case cracking could be a long-term issue for some lithium batteries. Fire within the battery can result from hot-spots due to manufacturing or material quality issues.

There are several types of lithium-ion battery available. Six of the most popular include Lithium Cobalt Oxide (LiCoO2), Lithium Manganese Oxide (LiMn2O4 or “LMO”), Lithium Nickel Manganese Cobalt Oxide (LiNiMnCoO2 or “NMC”), Lithium Iron Phosphate (LiFePO4), Lithium Nickel Cobalt Aluminium Oxide (LiNiCoAIO2) and Lithium Titanate (Li4Ti5O12). UPS manufacturers select the most appropriate lithium battery for their UPS designs.

For the actual installation, the choice over lithium or lead acid batteries can once again be dependent upon rack power density. The larger the UPS the higher the cost of the device and its associated battery technology. Whilst lithium batteries provide a far greater power density per square metre and provide for rapid charge/discharge cycling, they are up to 30% more expensive than lead acid. This increase in capital investment should be assessed over the lifetime of the battery. Lithium battery design lives are 15-20 years. Lead acid have 5 or 10 year design lives on average.

Where there is limited space within a server rack, the compact power density of a lithium UPS battery can provide ideal. Especially for installations under 10kW and only requiring a short duration runtime to cover a generator start-up. This may in the future lead to a wider adoption of lithium UPS systems within data centre environments, where general safety concerns are overcome. For smaller server room environments, lead acid batteries may remain the primary backup energy source for some time. Small server rooms tend to require larger battery runtimes because they do not have standby generator support. They may also lack a room or rack-level fire suppression system.

More info on lithium UPS batteries:
https://download.schneider-electric.com/files?p_Doc_Ref=SPD_VAVR-A5VTU5_EN

Rack Power Surveys

The objective of a rack power survey is to determine the size of the IT load, either in kVA or kW. The resulting information can be used for capacity planning, sizing a UPS or cooling system or a refresh of the overall server room or data centre facility. When our project engineers carry out a rack power survey, their initial approach is based on that for sizing a UPS system.

Uninterruptible power supplies have typically been sized in VA (volts x amperes) or kVA if the resulting number is greater than 1000 (k). VA and kVA measure the ‘Apparent Power’ required by a load. The ‘Real Power’ is measured in Watts (W) or kilowatts (kW).

Apparent Power and Real Power are connected by the Power Factor of the load. VA x Power Factor = Watts. The ratio is a number between 0 and 1. The switch mode power supply within most physical servers have near to 1 (Unity) power factors. The more closely the output of a UPS system is matched to the load the greater the energy efficiency. For this reason, modern UPS tend to be rated with Unity rated outputs where VA=Watts or 0.9. Older UPS designs may be rated at 0.8 or less down and even down to as low as 0.6 and this leads to a difference in the VA and Watts they provide as shown in the table below.

So, to size a UPS for a server rack installation, the total rack power draw must be calculated or measured, during a rack power survey. There are several ways to measure the power drawn by a server rack (or cabinet aisle) and the IT equipment within them:

1. Metered or Smart PDUs: if the rack uses metered or smart/intelligent power distribution units it should be possible to obtain the overall VA or Watts or Amps and Voltage being supplied via the PDU outlets. The PDU may have a meter built-in or be capable of remote reporting via a network SNMP card.
2. Sub-distribution Board Measurement: an electrician may be able to connect an RMS meter to individual branch circuits powered by an LV board within the room. This is only possible if each server rack or a set of comparable cabinets is supplied via a breaker. Great care must be taken if this is done with ‘live’ power loads to prevent inadvertent tripping of the breaker and for this reason it is not recommended.
3. Power Cord Clamp Meters: clamp meters use induction cores or coils to measure the amperage and voltage flowing through a piece of electrical cable. The cores or coils can be connected to an RMS meter or dedicated measurement unit for the recording of individual power cord measurements. Again, great care must be taken when attaching the measurement devices to the power cords as they are live and can be supporting critical loads

Sometimes it is necessary to use a combination of the above methods to determine the load power demands. For some devices it may not be feasible to make measurements. A typical example is a small network peripheral powered by an AC/DC adapter. Here the overall rating plate information could be used.

Once the overall power demands have been determined it is relatively easy to size a UPS system and determine the most appropriate approach in terms of a centralised or decentralised power protection plan, form factors (tower or rack mount installation), N+X redundancy and scalability factors, monobloc or modular, and the most suitable battery type (lead acid or lithium).

The total kW usage of the IT load also factors into the choice and sizing of the cooling system required, along with other installation related factors.

Server Rack and Computer Room Air Conditioning Systems

The biggest operational expense for any server room or data centre environment is typically that for electricity and in particular the energy required to run cooking systems. The greater the rack power density (and number of racks) the greater the demands on the local cooling system.

Rack power densities above 20-25kW may be more suited to liquid cooling system. Between 10-20kW per rack in-row precision cooling or underfloor pressurised cooling with a CRAC unit will typically be adopted. Below 10kW and where there is only a small number of racks, such as in a server room, a wall mounted air conditioning system will be used.

Whichever type of cooling system is installed, there will be a need for some form of redundancy and monitoring to prevent a sudden loss of cooling resulting in a catastrophic heat build-up. Once again, the overall rack power density (and number of racks) will make a major contribution to the overall choice in cooling technology.

Summary

Whilst rack power densities have not increased on average to the levels predicted in the last five years, the overall average size is increasing. For a data centre an average of 8.4kW still represents a large amount of power to be supplied by a UPS system and PDU arrangement and a large amount of heat to funnel away from the rack. At the rack level there are several UPS options which can be scaled to meet future expansion.

At the rack level, precision cooling choices are more limited and dependent upon the overall data centre design. Fan trays can be fitted to server racks to assist air flow but as important is some form of rack temperature monitoring to help prevent hot-spots and the potential for heat-damage to rack mounted devices.

How much high rack power densities will rise in the future will be dependent upon Moore’s Law and future breakthroughs in microprocessor design.

If you are interested in a rack power survey or would like a server room or data centre design review, please contact our projects team.