Blew Technology

Recently I’ve been setting up a pair of machines for web hosting duty.  As a part of my high availability strategy I’ve decided to mirror a Linux LVM Logical Volume from each host to the other.  This will allow me to fail services between hosts without losing any data with only a momentary blip while things shuffle around in the background.  Enter DRBD

In a nutshell, DRBD allows you to mirror (both synchronously and asynchronously) block devices over standard network connections.  These mirrors are flexible including features like “Truck Based Replication” (for pre-seeding a host locally, loading it on a truck and syncing it back up at a remote site) and split brain detection.  It also looks to integrate well with things like Linux-HA and Pacemaker.

In order to get started, you first have to set up a resource file.  The default global DRBD configuration options were pretty sane for Fedora Core 13, but there are a couple things worth noting:

/etc/drbd.d/global_common.conf:

• global
  • common
    • protocol C; – This indicates a Synchronous mirror. You can read about the other protocol options here
    • pri-on-incon-degr – These commands are run if the node is primary for a DRBD device, degraded and the data is inconsistent. Default behavior is to call notify.sh, which emails an appropriate error message, emails another messages notifying the sysadmin that the host is about to reboot and performs an immediate reboot without shutting things down cleanly through the shutdown command.
    • pri-lost-after-sb – These commands are run after the host loses a split brain election.  The defaults here result in the same ungraceful reboot as pri-on-incon-degr.
    • local-io-error – If the local IO subsystem returns an error the host is halted immediately without a clean shutdown.

It should be noted that the error conditions above will rely on the filesystem’s recovery mechanisms to maintain data integrity.  These events are serious enough that the underlying host may already be in a bad enough state that a standard “shutdown -r/-h now” wouldn’t return in a reasonable timeframe.

Getting Started

Now that you understand some of the more important defaults, let’s dig in and define a DRBD resource (or mirror).  This configuration will have to be put on both machines:

resource r0 {
    device          /dev/drbd1;
    meta-disk       internal;
    disk            /dev/volumegroup/logicalvolume;

    syncer { rate 50M; }

    on host1.blewtech.com     { address 192.168.2.1:7789; }
    on host2.blewtech.com     { address 192.168.2.2:7789; }
}

The addresses above are using network interfaces that are dedicated to DRBD.  If you’ve got fast drives you’ll probably even want to bond multiple Gigabit Ethernet interfaces together to maintain disk level performance. For my purposes I’ve set the syncer to rate limit itself to 50M in case there are other services utilizing the back channel interface DRBD will be using.

Now you can get your resource ready for use:

drbdadm create-md r0

And finally bring the resource online:

drbdadm up r0

Your resource is now online and ready to go.  You can get the current status of drbd a couple different ways:

service drbd status
drbdadm status 

If you’ve run this on both hosts you’ll notice that both hosts have the resource in a secondary state.  You’ll need to synchronize the mirror in order to make it usable:

drbdadm -- --overwrite-data-of-peer primary r0

Since we set a maximum sync rate of 50M in the DRBD configuration modern faster drivers will likely be held back.  We can tell DRBD to temporarily speed this up:

drbdsetup /dev/drbd0 syncer -r 110M

drbdadm adjust resource

Other Useful Tidbits

If you created the device but now need to fail it over to allow the other host to write to it, you have to first demote the primary host and then promote the other host:

Old Prmary:
drbdadm secondary r0

New Primary:
drbdadm primary r0

Performance with DRBD is good as far as I’ve been able to discern.  Unfortunately I don’t have enough spindles handy to try pushing beyond a single GigE link.

Hopefully this article encourages you to give DRBD a shot.  Once you’ve got it in your arsenal I’m sure you’ll think of all sorts of interesting problems to solve with it.