Following a recent summer storm, one of our customers experienced a roof failure on their building which resulted in their premises being flooded. Unfortunately, in this case the flood of water went straight through the middle of their comms cabinet. Below is the story behind what happened next.
When our client got into work that morning, they first noticed that their internet phones were all powered off. Upon further inspection they found that the computers could not access their server or the internet. Staff then went to the core of the business network (server room) and to their horror found that the room had experienced a flood. At this point they contacted Hobb Computer Services Ltd to seek urgent assistance and started the clean-up of water.
Being a managed business support client, our engineer was onsite within 30 minutes of being contacted by the client.
Upon arrival it was clear that the flood damage could not have occurred in a worse place. The comms rack that contains network switches, internet connectivity and the on premise / virtual servers had all experienced the brunt of the flood.
As business critical services for this client are mostly web based; it was agreed that the priority was getting the phones and internet services back up and running asap.
Some of the devices (switches, leased line, etc) in the rack were found to just have blown fuses but the Cisco managed switch at the top of the rack was determined to be totally dead which was the reason for all phones being powered off as they drew power from this switch via POe+. Using one of our maintenance spares for business clients, we fitted a temporary switch so that we could get business critical phones up and running.
With the phones up and running and connectivity re-established to the internet, the client could mostly function as a business and begin to take orders again. Attention was then turned to the on-premise servers.
The rack mount HPe Proliant server had taken a big hit from the flood. The server exhibited no signs of life and it was clearly visible that water ingress had occurred at least around the hard drives that sit at the front of the server. The decision was taken to remove the server from the rack and at which point, water flooded out of the server and soaked our engineer. Not what we wanted to see but we returned the server to our premises for further assessment.
Upon inspection we found that the server main board had experienced severe damage and that the hard drives were all soaked. The server was deemed to be beyond economical repair and would have to be replaced.
Using our HPe Partner status and distribution channel partners, a server was ordered for next day priority delivery.
Upon arrival the server was assembled and updated with the latest Microsoft updates. Our engineers reinstalled the Clients Veeam Backup & Replication software and set off a server restoration ‘job’.
Arriving back in our office at the early hours, the restoration was tested and all was found to be good to go. The server was returned to the client’s premises and fitted to the comms cabinet along with a replacement APC UPS that was also found to be flood damaged.
The servers were ‘booted’ and connectivity tested to Sage Accounts, Payroll, printing and their client data. The time and attendance clocking machines were polled and successfully downloaded historic data for the time they were down.
Timeline
Day One a.m.
- Call to arms
- Onsite to assess damage and fit temporary loans
- Business-critical phones and all internet connectivity/business web apps restored
- Server returned to our offices for assessment
Day One p.m.
- Server assessed and deemed to be beyond economical repair
- Replacement parts ordered
Day Two a.m.
- Replacement POe+ switch fitted to cabinet to power remaining phones
- Server arrives and is build to spec
Day Two p.m.
- Restoration of client backups
Day Three a.m.
- Verification of server restoration
- New Server installed at client’s premises and full connectivity restored
A highly successful recovery from what could have been a business affecting event.
Ian Hancock – Operations Director
“A massive thank you Hobb Computers team for the instant response at 7.30am to what was a potential disaster for a waste and recycling company that relies heavily on our cloud based systems including telephones, within a very short period of time we have half our system back to normal, new parts including a new server ordered and installed within 24hrs, all hands to the pump from Pete Harbron and his team saved the day.”
