Mid Essex Hospital NHS Trust Case Study

Mid Essex Hospital NHS Trust Ensures Business Continuity Clinical Services with Infrastructure Support From on365

The Client

Mid Essex Hospital Services NHS Trust (MEHT) based in Chelmsford has taken steps to ensure complete business continuity for its clinical IT services, implementing “n+1” level redundancy in its primary and back-up server rooms. The new platform, completed on 12 December 2008, was built with APC infrastructure consultancy and hardware from on365, which is a specialist in the planning, installing, management and optimisation of physical IT infrastructure and utility services.

MEHT decided to build in complete uninterrupted power supplies (UPS) and cooling redundancy in the data centre after experiencing several power losses at its Broomfield hospital site in Chelmsford. Although the trust’s IT and estates departments had maintained clinical services using generators and UPS – for a 14 hour period on one occasion – IT director Mike Casey wanted business continuity that would virtually eradicate the risk of any system downtime in any power loss situation.

This ‘safety first’ focus comes from the IT directorate’s operational responsibilities. It supports 43 different clinical services for 4,000 patients and 110 hospital consultants. Services, from orthopaedics to outpatients, including 20-30 separate departmental clinics daily, are organised through a real time Patient Administration Service (PAS). Any server ‘down time’ creates a time lag in delivering mission critical messages, potentially undermining patient care.

Mike Casey outlines the trust’s expectations: “System losses of more than a few minutes could force clinical staff to use manual processes, affecting patient care. Staff naturally expect an ‘always on’ service. If there is any interruption, IT would receive a massive backlog of messages, slowing services. There are a lot: pathology tests alone generate 2,500 messages an hour.”

The Challenge

The trust’s tender specified a resilient UPS and cooling platform for all clinical systems with a live environmentmaintained during the server rooms refurbishment – 160servers in 14 server racks with failover to a complete back-up generator and UPS system. Servers had to be maintained at optimum temperature through the replacement of the existing air conditioning plant and required the installation of underfloor cooling for non purpose-built server rooms.

After evaluating a range of tenders, Mid-Essex selected on365 and the Data Centre Consultancy as the organisations which would provide the upgraded infrastructure and begin work on the new system in Jan 2008. A key part of the tender process was finding a supplier that would adapt to the trust’s business needs and flexibility to work in a live server environment.

Mike Casey said: “Our business continuity tenders are very clear. We specified no down time – effectively to an end user, a potentially major server room failure would appear to be nothing more than a momentary screen freeze. As a result, these needs demand the highest specifications from our suppliers: the trust has high availability expectations and is working to achieve 99.9% availability on some of its core business and clinical systems. These tenders also seek clear evidence that a supplier is aware of new operational and technological developments and can prove the system’s integrity through testing.”

The IT team wanted insight as well as design skills from any supplier. As Mike Casey explained, “It was immediately clear that on365 and Data Centre Consulting’s engineers had a lot of experience and deep technical knowledge. They were familiar with innovations that ensured ‘n+1’ operations.”

The Solution

IT operations manager Barry Stannard’s in-house team, together with engineers from on365 and Data Centre Consulting, designed a complete primary server room and secondary business continuity room at the Broomfield site. This was based on:

Maintaining live environment throughout system implementation
Centralised management platform – giving visibility of all APC equipment
Deployment of server clusters with maximum density in available space in a non purpose-built facility and server rack testing
Assessment of power demand – assuming complete loss of power supply to servers and air conditioning units
Provision and installation of back-up APC UPS and battery units to support servers while reducing energy consumption
Before and after inspections to track progress and ensure the installation’s integrity
Business continuity testing schedule

on365 engineers designed a physical UPS infrastructure that provides complete failover, while Data Centre Consultancy came up with a cooling solution designed to provide optimum cooling for an older server room with limited space. The infrastructure also controls the level of power loading on back-up UPS systems, reducing component wear and tear and possibility of system burn-out. The server and back room were also given a “free air cooling” system based on a mix of hot and cooler server operations as well as the replacement of air conditioning plant with under-floor cooling.

For the main server room, on365 supplied and installed server racks, APC 80KW n+1 UPS unit and an APC battery frame and power distribution unit (PDU) ensuring uninterrupted power with reduced energy use. For the back up business continuity room, on365 provided and installed an APC 70KW n+1 UPS in an 80kW frame with fully populated battery frame and power distribution unit (PDU) rack with output breakers for all configured items to combat power surges. On365 is checking the server rooms’ energy use through an ongoing environmental monitoring service.

Chris Smith, marketing director of on365 commented: “The new business continuity platform ensures resilience, no down time and more energy efficient operation of back up UPS, which is becoming an additional cost consideration, even in mission-critical work of this type.” Back-up systems bring additional considerations as Chris Smith explained: “For a high density server installation, we provide UPS units designed to handle unity or leading power factor loads (modern high density blade servers

draw this type of load profile). On many UPS systems this type of load cannot be supplied without the risk of overload or failure of the UPS itself. This is often overlooked as it appears to be adequately specified, but not for a concentration of blade servers. The only other alternative is to significantly over size the UPS but this brings an unnecessary cost increase.”

The Outcome

MEHT now has a fully resilient server room with clear operational benefits too. IT director Mike Casey said: “We wanted to build a ‘heartbeat’ connection between the two server rooms. The new server rooms run more effectively and the design enables part or all of the system to be tested through complete business continuity testing, not ad hoc disaster management. Testing isn’t about a quick failover test either. We recently carried out a three-day complete failover for the Patient Administration System which included the restoration of all transactions from the new back-up system.”

Mike Casey adds: “Now we have complete redundancy, we can have different conversations with clinical colleagues: We can improve the way information is supplied to departments and help them improve patient care. We can only do this if we have a resilient and dependable IT Infrastructure. on365 helped to achieve that goal.”