Always have a backup plan

No one wants to plan for failure, but doing so is essential when working with technology. Over the past 30 days I had the pleasure of having two catastrophic failures. Working in healthcare where were errors can have life changing or ending consequences perhaps that term might not be fully correction. However, the failures were unrepairable, which if you have ever had one is not a fun time.

The first experience was when a database error took down the Emerging Health Leaders website. Not an enjoyable thing to discover. When I called the hosting company they were not much help. Thankfully I had installed Backup Buddy, one of the most important plugins if you have a WordPress site.

Now, given that I’m a nurse and not an IT specialist I did pay the hosting company to repair it. They said it would take 24 – 78 hours. Inside that time I worked with the backup buddy plugin, which had saved copies of the site to Dropbox, and had the site repair before they responded to my support ticket.

Then two weeks ago I had an single drive on my external storage device die. The device is where I store client, photography, website and video work due to the size of it all. It’s designed to be redundant.That way when one dies you replace the single drive let the drive repair itself and then you continue working.

So when the first drive died, I didn’t panic. I simply shut down the drive, bought another drive and waited. When it arrived I placed it in the Pegasus Promise R6 and set it to rebuild. Before it had rebuilt (it got to 63% after 4 hours) all the hard drives had managed to fail. This corrupts the redundancy and renders the files lost.

Broken pegasus promise r6

Now I panicked. Fortunately I checked my online backup through Backblaze, and discovered that I had 99.9% of my files backed up… all 2.5 Terabytes (that is a 2,500 gigabytes). Now if you have normal internet access downloading that would take a long time, and would over run most bandwidth limits. BackBlaze does provide an option to have a drive made and shipped. So within an hour I had requested this service and had my files back in 2 weeks. Now I just need to figure out an external storage solution.

These stories end well for me. I’ve heard many stories from friends that did not end well from them. From lost assignments before midterms, client work destroyed, or their entire portfolio of work lost when they had a laptop stolen. I hate these stories, and often feel sad. So before you even start your next project, please make sure you have a solid backup plan.

There are many affordable options, which are more than reasonable before hand. Remember you are paying for peace of mind, because once you lose all that work there is generally no price you wouldn’t consider for being able to bring it back.

3 Responses to “Always have a backup plan”

  1. If I had to guess, depending on diagnostics, there was either a short that burnt out the hard drives (assuming that they are individually fried), a flipped bit caused corrupting info to be written to all the drives (assuming everything works minus the raid configuration), or the raid controller itself fried. Regardless, if you did not have the cloud backup, you would still be able to get the data off the platters from a data recovery company. Usually it’s fairly easy (for them) to recover – remove the platters/board (if it’s an electrical problem), put them in a new drive, and fire it up. In the worst case they would have to try their best at recovering the data by scanning the platter sector by sector. That happened to me four years ago, which for a 500gb costed $1,500, and took 2 weeks for a full scan (my drive was in bad shape).

    Cloud backup seems to make sense for most people – the raid controller itself is a single point of failure, as well as the room it’s located in (flooding, theft, etc.). A cloud syncing drive is probably the cheapest choice for getting a real-time backup that’s in a different location. Though the drive should be encrypted.

    • Wow, $1,500 for 500 Gb that hurts. I would not want to pay to repair 2.5 Tb. I’m glad I had the working back up!

      The question I still don’t know is what to do with the drive? Should I just get rid of it, is it worth repairing, or should I try any of these old drives in a new system? Would appreciate any insight.

      • That was four years ago, so it would have been equivalent to around a 2tb drive today, and a full scan instead of a platter/board swap, which is what typical recoveries would be. Those would be more like $300-$600.

        The thing to keep in mind is that while raid decreases the chance of the loss of all drives, it increases the chance of the loss of one drive, which has to be replaced. So in a 6 drive raid, assuming $100 per drive, then statistically speaking, compared to having no raid at all, you would expect to spend $600 replacing the drives as they fail by the time the unraided single drive fails. If you can recover the failed unraided drive for $600, then the raid controller would be uneconomical and only useful in the case of requiring the data to remain available at all times. While I would have to work the numbers, my guess is that the most economical option for making your data fail ready would involve a smaller, 2-4 drive raid system (mainly for availability), a real-time cloud backup (for location based failures like short circuits, theft, fire, etc.), and archiving (for cases of accidental deletion).

        As for the old drives, given that they all failed at the same time, it’s likely not a mechanical problem. The likely culprit is the raid controller itself – either it has failed itself, it fried all the drives, or it made an error and corrupted the configurations. In the first and last cases, the drives should be fine. I would test the hard drives on a new system, and if they seem to work fine I would keep them.

Leave a Reply