Ugh, what a day. Started out not bad, went in early (leave home at 6:30 get to work between 8 and 8:30), and the traffic wasn’t horrible. One of the projects I’ve been working on has been some data storage for clients, basically a data vault. Anyway, in my tweaking and twiddling of some settings a few days ago, one of the client servers wasn’t backing up properly to our system. Not a big deal, except that the way that we do daily rotations meant that about 50G of data had accumulated that needed to be transferred to our system. Not a small task for a client with a DSL feed.
Ok, not a big deal, drive to the clients (about 16km away, about a 15-20 minute drive) pick up the disk and transfer it at LAN speed to our backup system. I’d done this before when the data was first seeded into the system.
I get back and plug in the disk to my system, and using it as a virtual disk to a VMWare virtual Ubuntu install boot it up. Look at the disk and say to myself “hmm…. why isn’t the size of the disk plugged into my system the same as that of the running client server I just left.
Hmm….
Here comes the part about Linux software RAID I didn’t know. When I pulled the disk out and restarted the system when the data was seeded a couple of weeks ago the RAID had worked perfectly, coming up in a degraded state. I had assumed that when I had put the disk back in it had seen the disk was there and started rebuilding.
Not so. The system never realized that the disk was back in and had continued merrily running along with only one disk.
So now I had to go back to the client, put the second disk in, let it resync (luckily only a 20 minute task), drive back to work, sync the disk to our local system (a several hour procedure) and then at around 3pm (incidently around when the traffic starts in downtown Vancouver) drive back again to put the disk back. This time adding it back into the RAID array properly.
End result, close to 2 hours to get in and out of work plus in and out of Vancouver 3 times totally another hour or so of driving. Bleah.
Bleah bleah bleah.
Needless to day I was not a happy camper. My fault I admit, totally my screw up… still not happy about it 🙂