Firewalling VMware ESXi with Smoothwall

VMware ESXi (also known as VMware vSphere Hypervisor) is a very popular virtualisation platform. VMware give ESXi away for free but it has no firewalling. I wanted to add a firewall and chose to use Smoothwall Express.  There are several other how-tos out there but I thought I would share my experiences with you too.
Continue reading

Posted in Uncategorized | 4 Comments

Software RAID1 plus GRUB drive replacement

Help! A drive in my software RAID1 array has failed and needs replacing. There are already a lot of posts on the web on how to replace the drive (e.g. here and here). Unfortunately, when I came to follow them, I discovered that the drive that I had as a replacement was a few megabytes smaller than the one I was replacing. This is a “bad thing, TM” as it means you cannot mirror onto it. Fortunately, LVM came to the rescue. Although the whole disk was partitioned, I was not using the whole disk so I could do some LVM trickery to allow me to reduce the size of the partitions and mirror them successfully on the smaller drive. I also made sure that I left some empty space at the end of the drive in case this happens to me again.
Continue reading

Posted in Linux, RAID | Tagged , , , | Leave a comment

Is Gluster the answer?

So, you don’t want all the hassle of setting up DRBD and Red Hat Cluster services but still want to have a replicated file system you can share between nodes.  What can you use? Well, Gluster is one option.  I tried it here and the set-up and configuration was easy.  Gluster uses FUSE (which is old in RHEL/CentOS 5, hence lack of mmap support).  As Gluster has a regular file system underneath tasks such as resizing the file system use standard linux tools.  Installation and configuration of Gluster are well-documented in the Gluster documentation so I won’t go into details.  However, there are some interesting “features” to be aware of, although these may be fixed in future releases (I was using Gluster 3.2.1).
Continue reading

Posted in Synchronisation | Leave a comment

Recovering DRBD+GFS after node failure

Help! I’ve set up DRBD plus GFS2 like you said and now a node has failed! Where do you turn to? Fortunately, the DRBD user guide has a whole chapter on troubleshooting and error recovery. Here is a procedure that worked for me when a node temporarily failed. What you can actually do depends on what the failure mode of the node was.

To start with, you need to fix any problems with the node and check that it comes up OK. Assuming that the DRBD device on the surviving node is OK and contains a good copy of the file system, you can put the recovered node into secondary state and re-synchronisse the device. On the recovered node, run:

# drbdadm secondary r0
# drbdadm -- --discard-my-data connect r0

If /proc/drbd shows the surviving node’s state as StandAlone then on the surviving node run:

# drbdadm connect r0

The synchronisation should start. Once it has finished, you can promote the recoverd node to state primary, activate the volume group and restart the gfs2 service:

# drbdadm primary r0
# vgscan
# vgchange -a y drbdvg
# service gfs2 start

You should have your gfs file system back.

Posted in Synchronisation, Troubleshooting | Leave a comment

Replicating block devices with DRBD

Creating a hardware setup for a shared file system with concurrent access has long been the realm of expensive SAN/NAS solutions. Now, it is possible to do this for yourself with only two PCs (I was using two virtual machines running CentOS 5.6). I was looking for a way to keep a file system in sync between two machines and wondered about replicating at block device level.

DRBD (Distributed Replicated Block Device) seemed to be a suitable solution. DRBD replicates a block device to another machine over the network and is included in most recent Linux distributions. As it is replicating at a block level, it does not need to inspect every file and therefore does not suffer the same performance problems that Unison suffers (see earlier post on Unison). DRBD is particularly useful when updates to the shared device must be seen immediately by the other node.  You can treat the DRBD device like any other block device, using it to create file systems, as swap space (shudder) or for some other exotic use.

As my requirement was to mirror a file system with concurrent access, I had to build a shared file system on top of DRBD and chose GFS2 which will provide the locking required for concurrent access. This allows two machines to have redundant storage mirrored on both hosts. In the end, I didn’t use DRBD plus GFS because I was worried that problems on one node in the file system layer could corrupt the file system and DRBD would happily mirror the corruption to all nodes in the mirror. Read more to see how it was all set up. Continue reading

Posted in Synchronisation | 6 Comments

Uniting directories with Unison

How do you synchronize two directories which are both changing? Rsync is ubiquitous and has been the staple of sysadmins for a long time. Unfortunately, it cannot differentiate between a file deletion on one node and a creation on another so two-way synchronisation is difficult if not impossible.  Just search Google for it to see how many scripting solutions there are! Fortunately, there’s a solution: Unison.

Unison has many advantages over rsync (read the web page for details, I am not going to repeat them all here). Importantly it can tell the difference between a deletion on one node and a creation on another as it holds information about the state of the synchronised folders (AKA replicas) in a database file in your home directory. Unison also detects conflicts when a file has been updated in both locations and provides a mechanism for automatically or manually resolving these.

So, how do you use this whizzy piece of software? Well, the documentation is very good so again I won’t duplicate that. Use of Unison is very easy and really does just involve installing the RPMs (available in the EPEL repository) and then running the command according to the documentation.  To make the commands simpler, you can set up profiles which contain all the configuration options you want to specify.

So, Unison is easy to install, easy to run and is well documented.  “What’s the catch?” I hear you cry. Well there is one that for me was a show-stopper. I was trying to syncronise a file system that holds over a million files. As Unison needs to traverse the file system for changes, its performance starts to drop as the number of files increase.  I noticed that the performance was starting to suffer at about 800,000 files. This might be partly due to my low-spec virtual machines that I was using for testing but other replication solutions I tested did not suffer in this way. I left Unison running over a weekend and found it still had not finished when I came in on Monday! This is no good for my purposes.

For anyone struggling with with Unison and lots of files, there is a way of running Unison in batches that speeds things up. Begin by creating an Unison profile (in this case ~/.unison/synctest.prf).  It should have the following contents:

# Unison preferences file

# root of synchronisation
root = /unison
root = ssh://remote-host//unison

# don't prompt
batch = true

# diff command
diff = colordiff -u CURRENT2 CURRENT1

# ignore lost+found
ignore = Path */lost+found

Once the profile is set up, you can run Unison in a shell loop for each of the top-level directories in your replicas:

for topdir in <list of directories>; do
  unison synctest -path $topdir

This took 2 hours rather than several days to synchronise the replicas.  This was too slow for me as I would like my replicas to be only 30 minutes out so I had to rule out Unison for my use.  You may find that it suits your situation perfectly though.

Happy syncing!

Posted in Synchronisation | Leave a comment

James’s storage ramblings

I’ve recently started in the HFS Team at OUCS. To share what we learn and do, I hope to put information about various storage technologies here for others to see.  Note that these are my own views and not those of Oxford University Computing Services or the University of Oxford.

Posted in Miscellaneous | Leave a comment