You wonder oh god, who is emailing me, who is visiting my website and further the challenges of how can I do business without my email and website. It’s amazing how the world as changed to level of everyone being so lost without it.
One of my Companies, Cityhosting had the unfortunate situation of a Firmware update that took one of the servers offline for a few hours. It was the second time in less than 6 months that I’ve had this happen. Hosting companies promise 99.9% uptime. I’m now in the 60% uptime based on averages over a period of a year. Not something I’m proud of, but it did force my company to “think” through this crisis (after it was fixed) to how do I make sure that this doesn’t happen again.
We analyzed the issue and determined why this happened. First time was a power outage and the generator that was to power the second servers (backup didn’t come on). This time it was an update in response to the Heartbleed Bug that was out there. The situation resulted in downtime for clients, and my business included for hours.
Out of every issue you can find a solution, or the problem can lend itself to a better situation. Here’s what I’m doing to make it not happen again.
1. Moving all servers back to Canada. I currently have 3 in the USA and 1 in Canada. The main reason is that I can have things closer to us; but also for the fact that it’s safer and protected in Canada.
2. We are putting in a notice to clients that will be notified by TXT message of where to go to get updates when the server (could) be having updates or might be down for maintanence.
3. Redundancy (RAID) Redundant Array of Inexpensive Disks. This just means that the servers in Vancouver will kick in when Toronto is down.
So, I’m chalking this situation as a costly learning lesson to the things that we need to do in business to ensure that they are working better. I was too comfortable with our setup and processes and now it’s time to revisit not just this one, but many others. I guess this was a good thing…in a bad sort of way!