Catastrophic Data Loss: Why C@C is Whack

Posted on 29th Oct, 2017 in Uncategorised

There is a golden rule in IT. Always have backups.
However, not everyone follows that rule and despite working in the same field, I made that mistake.

What follows is a story of how buying things that are too good to be true and then being lazy laid waste to my website.

How It All Began

It must have been 2 or 3 years since I first heard about Cloud at Cost (or as everyone I know now calls it "C@C" but pronounced "cack") . They were a fairly new hosting provider located in Narnia Canada that had gained massive popularity overnight. What made them stand out from all the other hosting providers is that they were selling Virtual Private Servers for a one-time fee. Whereas, the industry standard was (and still is) to charge a monthly or yearly fee. Despite having the experience to have known better, I decided I'd take a chance on them since it was cheap. If they actually lived up to their promises I'd be saving a bundle.

That Was My First Mistake

Things were great at first but I didn't really make use of the VPS I had paid for. Despite that Cloud at Cost kept putting things on sale, so decided I'd buy an even beefier VPS as well. Both continued to sit unused until I acquired the solidus.systems domain. Now I definitely had a reason to make use of these servers. I threw this very low traffic site, built on a very lightweight CSS & blogging platform on the first server and the database backend on the other. Within weeks performance started to degrade on the web server. The I/O latency was absurdly high... and then disaster struck. People started rebooting their servers to try and fix the problem but they never came back up. Instead many of us were greeted by kernal panics. C@C started putting out notices about how users would have to delete and recreate their servers if they already got that far. Yet this didn't deter me... even when I had to delete and recreate the web server. Fortunately, the database server was stellar and never experienced a problem.

Hit By A Car

Between the occasional mass kernal panics, came increasingly frequent maintenance to add more infrastructure. I guess that's necessary when you face explosive growth overnight but it never seemed to be enough to keep up with demand. Despite that, they'd hold server sales on an almost weekly basis. It almost sounds like a ponzi scheme; selling ridiculously cheap VPSes to as many people as possible in order to buy more infrastructure to serve your growing customer base. One day it will have to come crashing down but at this point the only thing down was the network. Somehow a car crashed in to the datacenter's fiber connection, bringing down everyone's network connectivity. Yet I still did not run screaming. Also lurking in the shadows was the fact that I had still forgotten to set up backups for my now frequently troubled servers.

Ransom

I suspect things eventually became too much for C@C and they decided it was time to start charging an annual fee for all those one-time payment VPSes they had sold. This way they'd lose customers/demands on resources while also gaining additional revenue. For reasons of convenience, I decided to stay aboard this sinking ship. This was the point of no return.

The Apocalypse

By July 2017 I had added applications to my servers which I relied on for more than just shitposting on this blog. One day, I noticed that I couldn't load the site and decided to log in. As usual, things were running excruciatingly slow, so I decided to give the web server a reboot. It never came back. I opened a ticket that was closed without resolution. Opened another to reopen the previous one. The answer I got was the same each time. "You'll need to delete and recreate your server" because it was lost. As in, they couldn't find or contact a virtual machine on the servers in their datacenter. WHAT THE ACTUAL FUCK?!

That was a major inconvenience, but fine, I can deal with having to recreate the web server (for what must have been the fourth time). Then as I had finally resigned myself to my fate, the same happened with the database server. Now truly everything was lost. Once again, I opened a service ticket. Once again, they claimed they had lost the server. Clearly there was an unthinkable level of incompetence sitting on the other side of the ticket desk and now I had damned myself to needing to rebuild everything from scratch.

So here we are.
Learn from my mistakes, always back up your systems.
Always have a back up to the backup.
Finally, DON'T USE Cloud at Cost. It'll only cost you in the end.