On Thursday, I asked the company that hosts Dark Castle's dedicated server to add remote KVM and power reset hardware to our server. This will incur an additional cost of $40/month on top of our current hosting fees. I requested this so that I can safely reboot the DC server without fear of an OS upgrade problem causing the server to go down where I cannot login to it remotely in order to fix it. With this new hardware we can stay more up to date with OS patches keeping our server safer from security vulnerbilities and other bugs.

When the technician who installed the remote KVM did so, he rebooted DC. He did not tell me that he was going to do so. I do not know if he did this safely or not since he didn't have password access to our server. After the remote KVM was installed, I could connect to it, but there was a problem keeping me from seeing the virtual monitor remotely. I reported these problems to the technician. While troubleshooting the issue, the technician rebooted the server several more times. He did not tell me he was going to do this and I do not believe that he did so safely. I think he abruptely powered off and on the server. I believe this caused the the filesystem to become corrupted.

While we use a journaling filesystem which is supposed to protect the server from incidental power outages, apparently multiple abrupt reboots one after another are enough to cause corruption. I started fscking (similar to chkdsk) the filesystem before lunch today and by the time I left work it still had not finished. Since the filesystem could not be fixed I had no choice but to reinstall.

We have nightly backups of the server which we keep on the server and occasionally I copy only the Dark Castle mud program and its related world, characters, etc offsite. Due to this, I did not have an offsite copy of the entire server which I could restore. I only had an offsite copy of the Dark Castle mud program and its related files. I reinstalled the OS from scratch and restored the forums and Dark Castle files from their backups.