Monthly Archives: August 2013

Prism vs LogBlock, misc.

I am testing the Prism block logging addon in minecraft; it has more advanced rollback features, and records more events, more usefully. So far, so good. Bit of learning curve, but once we’re over it, it should allow us to do some very helpful things, like previewing rollbacks before committing them, and recording what player killed what, or picked up what. Hopefully it prevents some arguments. I’ll have to rewrite a couple of auditing scripts.

Today we also had an interesting WorldEdit accident, one of the mods set a pos2 but left the pos1 back at the previous location, and then did a //set 0. Server locked up, and intending to force a rollback, I hit ^C on console. Unfortunately, that saved the changes that did go through, so in hindsight, I should have suspended java and did a kill -9 to prevent it saving the changes, an approach that has worked well in the past.

Oh well. Restored the missing areas from backup. Sadly, WorldEdit does not in the current build restore sign contents, so some shops in the mall got messed up. The dev build evidently fixes that.

BungeePerms is updating to use MySQL, which should allow me to set up tab list colours pretty easily. The main problem was generating a permissions.yml from a database export of PermissionsEx data, and I didn’t really care enough to go to the effort. But if they’re both in the database, then porting the data over is a relatively simple procedure, I can just make it part of the BungeeCord startup script or something, and it’ll sync once a day.

Server stats

I was having a conversation with someone, and the question of the server stats came up. Here we go:

The server hardware is actually two Dell R420 servers. From that beginning, I added a hardware RAID card, three rotary hard drives (1TB each), and one solid state drive (128GiB), 16GiB of RAM, and Intel Xeon E2430 CPUs, six cores each, 2.3GHz.

Two of the drives are RAID-1 (mirrored) with the third as a spare that will kick in automatically. I chose that configuration because a 3-disk RAID5 and a 2-disk RAID1 have very similar performance and reliability characteristics: they both have 2x read I/O and 1x write I/O, and can both tolerate one failed drive. With mirrored/spare, however, I trade off some space for automatic failover and rebuild onto the spare drive. I use the SSD as a block cache for the RAID1; that is, writes to the disk first go onto the SSD, and then get written out more efficiently, sequentially, to the rotary storage in batches, getting the best of both worlds. I use Bcache for that part.

The two servers both run Debian, a lightweight Linux distribution particularly well suited for servers. They are mostly identically configured; one is named ‘node1′ and the other is named ‘node2′. Installed on both servers is a piece of software named Ganeti, which manages KVM, a hypervisor; it’s used to run “virtual” machines, that is, machines that run virtually, separately from one another and can share a single physical computer. Ganeti lets me run VMs in a master-slave configuration, mirroring a VM between both node servers in such a way that if one server fails, the other server can take over. VMs can be migrated between the servers without people logged into the VM even noticing that it happened. For instance, I can move all the VMs onto node2, reboot node1, and then move all the VMs to node1 and reboot node2. Users don’t even notice, in theory. So, it’s pretty cool technology.

Right now, minecraft pretty much gets a node to itself. It’s extremely overpowered for the task, but I’ve specced things out to hopefully last me five or six years, so I’m pretty comfortable with the investment.

diskstats_latency-week diskstats_latency-week (1)

Hard drive latency graphs for the last week on my two node servers. Notice how disk response time goes from ~100ms to ~10 on both servers.

aio=native on KVM. Pretty great. Had to patch it into Ganeti.