So the recent outage showed off some of the limitations of our current setup. I’ll be taking the server offline for an estimated four hour maintenance window tomorrow to work on it. If you’re into geeky details, read on.
The current server is two CentOS 7 virtual machines, running inside a bigger server. One runs the database server, the other runs the Minecraft Spigot instance. So far, so good.
Both of these VMs are inside a Dell PowerEdge R610 server with 32 GBs of ram and about half a terrabyte of storage. Still good so far.
The hard drives can be paired for redundancy; putting drives into an array called RAID is standard practice for servers, as it lets you use a bigger pool of storage and can even handle it if one drive dies.
Here’s the lousy part – our current hardware can only support two drives being mirrored, called RAID1. This puts our database server on one RAID1 spot, and the Spigot server on the other one. One of those disks is failing, but the hardware isn’t tell me which one. So I need to pull those disks out and check them, to find the bad one.
Meanwhile, I have a fancy controller that lets us put all the disks together in a big pool called RAID5. This is faster, and lets us have all the servers in the same spot. Nice!
So if I’m going to pull the disks to check for the bad one, I might as well upgrade us to the new RAID5 fanciness.
Anyway, I’ll post more after I’ve done the upgrade. It’s all back-end stuff that shouldn’t affect players, but sometimes people are interested in this stuff. Later skaters!
While adding additional disk storage to the server today, I found quite a bit of disk corruption in a non-protected hard drive – basically, a problem with the server’s data. While I was able to catch it before we had a crash, it meant a 30 minute maintenance window turned into an all-day project. Phooey!
I’m working on restoring the server right now, including migrating out the failed disk and shifting things so our storage is redundant and protected. Sorry for the service interruption! We’ll be back online as soon as I can make it happen.
On Sunday evening – one of the lowest usage times for Hide Your Fire – I migrated our server on to new hardware. This is a big step up, letting us improve performance and giving us lots of room to add plugins and game modes.
As part of that process, I reactivated a few plugins that long time players here should recognize – Stables and SimpleClans.
I also switched out anti-grief tool CoreProtect over to a database system. This is really cool – it’s now using MySQL instead of a flat file. This makes it faster and lets us keep records longer. But I want able to import the old data.
If we need some older data for roll back, it’s available. But I’ll need to temporarily switch back to the legacy system to do it. If you need something restored, ask me directly. I suspect within a few weeks this will no longer matter.
We have a few more upgrades to go in the near future. I’m going to be installing faster SAS disks in the servers for speed and backups (RAID5, squee) but hopefully these upgrades will be transparent to players.
Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!