Server Room Audits and Data Centre Design Standards

When we are asked to audit a server room or data centre one of the first things, we want to establish are the design standards and management systems the facility should conform to. The answers to this question can vary dependent upon the size of the IT facility and to some extent the industries they serve.

Hyperscale and colocation data centres will typically require auditing to specific standards and levels of availability. Smaller on-premise enterprise data centres and server rooms may be more concerned with how best to solve a specific problem such as a critical cooling issue, how to implement greater power resilience and/or improver overall security and business continuity planning.

Global Data Centre Standards

Data centres have been around since the early 1950s, when computing companies such as IBM deployed their mainframe computers. At that time there were no specific standards for data centre design.

Over the decades the industry evolved with a reliance on standards for networking and telecommunications. In Europe, the principle standard for data centre design has been EN 50173, covering Information Technology and Generic Cabling Systems. In the US ANSI/TIA-942, the US standard for Telecommunications Infrastructure for Data Center, guided design and best practice. The issue for both standards is that they are cabling-based and whilst they have been updated, they do not specifically mention critical power and cooling system infrastructures – a main criticism of the latest edition of EN 50173-5:2018 which now covers Data Centre Spaces.

European Standards (EN) are documents ratified by three European Standards Organisations including CEN, CENELEC and ETSI. Recognising the need for a specific data centre standard has given rise to the EN 50600 series of standards for best practice and design standards in data centres, including power, cooling, telecommunications, and security (including fire suppression). The standard also covers operations and management and provides recommendations for energy usage optimisation, reduction, and sustainability.

EN 50600 Series for Data Centres

The EN 50600 Information technology. Data centre facilities and infrastructures. General concepts were launched in 2014 with EN 50600:2019 being the latest version of the standard.

EN 50600 is a set of data centre specific design standards that describes the general principles and common aspects of data centre design and operation including terminology, parameters, and reference models. Specific critical infrastructure systems are covered including power, cooling and security, lifetime operations and management including energy efficiency, business risk, operational cost analysis and key performance indicators (KPIs).

For a copy of EN 50600 see: https://shop.bsigroup.com/ProductDetail?pid=000000000030374947

The EN 50600 series was developed by CENELEC, the European Committee for Electrotechnical Standardisation. CENELEC is the European Union’s principle electrical standards authority but the EN 50600 series can be applied globally. This is because the standard has been designed to satisfy the needs of all member countries within the EU rather than at a single nation level.

EN 50600 has developed over time and the International Standards Organisation (ISO) has published several EN 50600 documents as ISO Technical Specifications for Data centre facilities and infrastructures including:

• ISO/IEC TS 22237-1:2018: Part 1: General concepts
• ISO/IEC TS 22237-2:2018: Part 2: Building construction
• ISO/IEC TS 22237-3:2018: Part 3: Power distribution
• ISO/IEC TS 22237-4:2018: Part 4: Environmental control
• ISO/IEC TS 22237-5:2018: Part 5: Telecommunications cabling infrastructure
• ISO/IEC TS 22237-6:2018: Part 6: Security systems
• ISO/IEC TS 22237-7:2018 : Part 7: Management and operational information

The EN 50600 series of standards therefore provide a solid basis against which to audit a data centre but there are other design principles that can be applied, even to smaller computer and server rooms. The most widely adopted of these is the Uptime Institute’s Tier-rating system.

The Uptime Institute Tier Classification System

The Uptime Institute is an advisory body in the data centre industry whose focus is on improving the performance, efficiency, and reliability of data centre facilities through collaboration and performance standards and certifications.

The Uptime Institute is taken to provide the benchmark when it comes to resilience in terms of power and cooling and their ‘Tier Ratings’. Though not a standard, the Tier-rating scheme allows data centres to be audited against specific Tier-levels from I to IV as to their power and cooling level resilience and redundancy levels.

For more information see: “System Availability and Tier-Levels”: /blog/data-centre-design-availability-tier-ratings
The Uptime Institute’s tier rating levels helps to apply a focus on to the critical power and cooling systems within a data centre and on their sustainability and availability in terms of the 9s. A Tier IV data centre offers four-nines (99.995%) availability and is fault-tolerant and concurrently maintainable.

Certification by the Uptime Institute is a multi-stage approach that includes design certification and then

• Tier Certification Design Documents (TCDD): consultants from the Uptime Institute review the design to ensure each fundamental subsystem (electrical, mechanical, monitoring and automation) meets fundamental design concepts.
• Tier Certification of Constructed Facility (TCCF): consultants review the completed facility to ensure that it has been built by contractors to the approved design.

The physical audit is important as actual builds can vary from approved designs and this can lead to a weakness being introduced into physical data centre infrastructure support systems. The TCCF is awarded for a two-year period as technologies change, as can data centre practices.

The Uptime Institute’s tier-rating system is a great way for a data centre to demonstrate both its design and physical building operations in terms of resilience and availability. The same principles can be applied to smaller server rooms and computer rooms when it comes to reviewing power (LV switchboards, UPS systems and PDUs) and cooling (air conditioning) arrangements.

Energy Efficiency and Power Usage Effectiveness

The data centre industry is sometimes quoted as the 5th utility’, alongside electricity, water, gas and telecoms, such is its size and importance to how we work and live. Energy usage and efficiency continues to be a major concern within the industry. Not just the amount of energy used by large hyper-scale facilities but the costs (overheads) for colocation datacentres which must be passed on to clients.

There are several ways to analyse and compare energy usage for both server rooms and data centres.
Power usage effectiveness (PUE) is a ratio developed by the Green Grid to measure energy efficiency in a data centre. As such it is the result of the electrical and mechanical design of the data centre and its implementation during the build-out and operational phases.

PUE = Total Facility Energy / IT Equipment Energy

The reciprocal measure of PUE is Data Center infrastructure Efficiency (DCiE)

DCIE = IT Equipment Power / Total Facility Power x 100%

More info: https://www.thegreengrid.org/

It is important to note that PUE does not factor in sustainability and how the electricity used to power the facility is generated e.g. nuclear or renewable power sources. In addition, PUE does not consider energy reuse. Some larger datacentres may reuse heat created during the cooling process which in turns reduces the total energy consumption of the facility.

PUE is a relatively easy energy metric to compute and can provide a useful benchmark for measuring energy efficiency improvements ‘in-house’ and for comparing to other facilities. In a carbon conscious world, PUE provides a metric that organisations and businesses can use to select data centre colocation suppliers.

The ISO is also evolving standards to look specifically at data centre energy metrics. ISO/IEC 30134 Data centres – Key performance indicators. The standard provides a definition of terms such as Power Usage Effectiveness (PUE) and covers nine KPIs in all including Energy Re-use Factor (ERF) and Water Usage Effectiveness (WUE), and IT efficiency metrics including IT Equipment Utilization for Servers (ITEU_SV).

General ISO Standards

There are several other ISO standards some of which are specific to the data centre industry which can be included for audit purposes and these include:

• ISO/IEC 11801-1:2017: Information technology – Generic cabling for customer premises – Part 1: General requirements
• ISO/IEC CD TR 21897.2: Information technology Data centres – Impact of ISO 52000 standards for energy performance of buildings
• ISO 50001:2018(en): Energy management systems – Requirements with guidance for use
• ISO 52000-1:2017(en): Energy performance of building – Overarching EPB assessment – Part 1: General framework and procedures
• ISO/IEC 20000-1:2011: Information technology – Service management – Part 1: Service management system requirements

ISO 20000-1:2011 can be applied by Cloud data centre providers for Infrastructure as a Service, Software as a Service and Platform as a Service provision.

Further more general ISO standards that can be covered during an audit including the ISO 9000, ISO 27000 and ISO 14000 series:

• ISO 9000 Quality Management: ISO 9001 is a recognised quality management system (QMS) certification. Its application in an IT environment can help to ensure documentation is available for the audit to progress efficiently in terms of non-conformance and incident reporting and corrective actions.
• ISO 27000 Information Security: ISO 27001 and ISO27002 certifies that an organisation has an information security management system (ISMS)and controls in place with relevant documentation and plans for business continuity and disaster recovery.
• ISO 14000 Environmental Management: an ISO 14001 certified environmental management system demonstrates an organisation’s commitment to the wider environment it operates in and to reduce its environmental impact. ISO 14001 and an organisation’s Environmental Management System (EMS) can provide energy usage and performance metrics that can assist a server room or data centre audit.

Most organisations with a substantial investment in their server room and data centre facilities have ISO 9001 as a minimum. ISO 14001 may also be a generic management system alongside OHSAS 18001 or ISO 45001 OHSAS (the replacement standard) for health & safety. ISO 27001 is important from an information security management point of view and may be essential for colocation data centres and as a requirement for suppliers in to cyber security aware industries such as utilities.

For more information see: https://www.iso.org/standards.html

Cyber Essentials Certification

Cyber Essentials is a UK government information assurance scheme operated by the National Cyber Security Centre (NCSC). The certification is the minimum required for organisations to take part in public sector contracts where there is the need to transfer public sector identifiable information. Organisations are assessed against five basic security controls and can be qualified asses for Cyber Essentials Plus certification.

ASHRAE and BICSI Standards

There are other specific standards that can influence a server room or data centre design. These include ASHRAE TC 9.9 2015 covering the ambient operational temperatures for IT equipment. ANSI/BICSI 002-2014 which is US-focused on data center design and implementation best practices and ANSI/ASHRAE Standard 90.4-2019, Energy Standard for Data Centers.

Summary

Which design standards to audit a server room or data centre against is often dependent upon the problems or issues faced by the organisation. The Uptime Institute’s Tier-ratings scheme is the easiest as it focuses on resilience and availability, principally in the critical power and cooling systems. Any weakness here could lead to downtime. Calculating PUE and comparing this to other organisations can identify energy efficiency issues that can lead to the need to improve cooling and power provision. An audit to EN 50600 is more comprehensive and can encompass a wider set of subsystems within the facility. Though a longer audit, the outcomes will be a more secure and resilient facility and one that can itself be certified to the ISO data centre standard.