Note: This post starts a bit technical, but even those less technical should benefit from the second half of the post.
Today I had the very exciting event of a hard drive dieing on one of my servers. Luckily, I had the drives mirrored and nobody noticed. The server just kept running off of the good drive. I’ve had this happen a couple times and it always makes me happy to say that we lost a hard drive and nobody noticed the difference. Nice to know that things work the way they should be working.
However, the thought of losing the other hard drive before I could restore the second drive is always a little bit tense. Luckily for me, I was able to just connect to the raid controller (yes I just lost half my readers) and rebuild the failed hard drive and we’re back in business (at least for now).
I admit that when this happened I went straight to my backups to make sure that they were working correctly and I immediately made a backup of all the databases on that server. Of course, the thought being that if I lose both hard drives I’d at least have my backups (which I’d saved to another machine).
Enough technical already. My point of this post was that this event really made me think about what a doctor’s office would do if they lost a hard drive. They better hope they have good IT staff to be able to fix it. It’s far from rocket science, but you do have to be careful in these situations. The other important lesson is who do you have monitoring that the drive failed? Most servers have the ability to notify you when a drive fails. However, how many doctors or IT people that support doctors have set up these notifications?
I wish I could say this was an isolated incident that’s rare. Over the past 4+ years in my current job, I’ve had 3 or 4 hard drives fail. Not a huge number, but also enough to require a good plan for when this happens. I’ve often heard that the first thing to die on a server is the hard drive. Why? Because the hard drive is a moving part (the disk spins) and moving parts break. So, you just have to expect for these things to happen and plan for it.
At the end of the day, nothing was lost and there was no down time for our EMR server. That’s the good news about a well designed EMR server. The problem comes when no one’s watching for these events or when the EMR server isn’t installed in order to plan for these types of events.
I’m pretty sure my SaaS EMR friends are going to love this post. There’s no doubt this is a huge plus for SaaS (hosted) EMR companies, but there are other challenges associated with SaaS that should be considered as well. I’ll let the SaaS enjoy this benefit for now and leave those comments for another post.