[Guest article by Dominic Hargreaves from the Systems Development and Support team]
Regular readers of the Networks team blog will know that preparatory work to enable IPv6 across the University backbone has been underway for some time. This article offers a perspective of this from the point of view of my team which is concerned with the server side of things (in particular GNU/Debian based). Some of this will be fairly closely tied to the bespoke system management infrastructure we use, but I hope it may be of interest in any case.
First, a bit of background. I’m taking it for granted (based on, for example, earlier postings on this blog) that readers are familiar with the basic notions behind IPv6 and the reasons to deploy it. My team provides and hosts a large number of network services, some of which are critical to the business of the University and which other IT services rely on (for example our Kerberos and Web SSO deployments) and this means we have some interesting trade-offs to be made. On the one hand, we would like to be in a position to offer our services over IPv6 as early as practical, to allow not only our services, but also those offered by other service providers throughout the University, to be available over IPv6. We don’t want to be in the position of holding providers back from their own deployments. On the other hand, for all the same reasons, we cannot risk introducing significant elements of service disruption or unavailability to our services. One current study puts client loss (systems who will be unable to contact a dual-stack service due to broken IPv6 at their end) at somewhere between 0.001% and 0.02%. For most applications this is acceptable, but we will need to pick our moment to enable some of our more critical services. My hope is that this figure will continue to fall as more OS vendors pick up on the various bugs and as embedded devices are replaced/upgraded, but for each service we will need to decide on the right time. In fact, this figure has dropped somewhat since I first drafted this article in December.
All of this means that although we might not be IPv6-enabling webauth.ox.ac.uk tomorrow, we need to start to become familiar with the fine details of offering IPv6 services now, so that when either we are comfortable with enabling our core services, or when there is external pressure to do so, we will be able to do so easily and with confidence.
A likely candidate
As it happened, in mid-October, I was putting the final touches to a new version of the OUCS blogs service (where this article is being published). This was the result of a project to ensure that the existing service, which was set up as a pilot, was put on a firm, managable footing. It’s also a service which, whilst useful to OUCS, is not generally considered critical to the business of the department. At the same time, I knew that the networks team were beginning to light up a few of their more minor services (IRC and the lightly-used webcache being two examples), and it seemed like an ideal opportunity to get some real experience in providing an IPv6 service without worrying too much about a small amount of client loss. I contacted the networks team, and following some discussion about the nature of the IPv6 backbone service (itself still very much a pilot) we agreed to go ahead.
At first glance, there’s nothing particularly complicated in deploying an IPv6 Apache service. Once you have the basic OS configuration up and running, all that really remains is to add the “Listen” stanza and to add the IPv6 address to the “VirtualHost” stanza (we tend to explicitly configure IP addresses for our vhosts) and throw your IPv6 client test platform at the service. Of course, the devil is in the detail…
Application support
For a web application which sits behind Apache (which we already know has good support for IPv6) there isn’t much to worry about with IPv6 support; in general it doesn’t know or care which is being used. Having said that, the client IP address is available to the application, and if it tries to store or manipulate that address, you do ned to check that it understands IPv6. This was easily checked with WordPress, which stores the IPv6 address of commenters in its database for later display to blog owners, and I didn’t find any bugs here (although I did later find that WordPress was confused by literal IPv6 addresses in URLs, a fix for which I submitted upstream).
Other web applications may vary, especially if they include support for things like host-based ACLs. It’s often worth asking your vendor or software project whether they have checked their application for IPv6 support in advance. And of course applications which listen directly on the network need much more fundamental support.
Debian networking configuration and multiple addresses
Our standard deployments make heavy use of service addresses – additional IP addresses used for specific services on a host. In IPv4 land, on Linux, this usually means using aliases. In many ways aliases look like completely separate network interfaces, and are configured as such. In IPv6 land, however, you generally add multiple addresses to a single interface instead. This meant some of our built in assumptions about how we configured things changed, and one problem here is that Debian (as far as I know) does not yet support the addition of multiple addresses to an interface in its standard configuration (/etc/network/interfaces). In practice this isn’t a problem; we make use of the facility to run other commands before an interface is brought up, and use the iproute2 tool to add additional IPv6 addresses to the interface.
One interesting problem I ran into when you have multiple addresses on the same network is being able to know what happens when an application makes an outgoing connection without explicitly binding to an address – which source address is used? With IPv4, the aliased interfaces are automatically deprioritised, and normally the non-aliased interface is used. This behaviour works well for us because it provides a degree of predictability. (In fact, we often want to bind to a particular aliased address for a given application, but this is either handled (hopefully) by configuring the application to use a given source address, or in some cases with iptables nat table trickery.)
With IPv6, however, things are different. If you add several addresses to a network interface, you’ll find that the default source address used is not always predictable, although in practice I’ve often found that the last address you add will be used. This is normally not very useful, but there is a fix, which is to mark the “additional” addresses as deprecated, via a command like:
ip -6 addr change <addr>/<mask> dev <interface> preferred_lft 0
so our interfaces configuration files now look something like this:
iface eth0.100 inet6 static address 2001:630:440:163::604 netmask 64 gateway 2001:630:440:163::1 pre-up ip -6 addr add 2001:630:440:163::602/64 dev eth0.100 pre-up ip -6 addr change 2001:630:440:163::602/64 dev eth0.100 preferred_lft 0
As I write this it occurs to me that it’s probably worth taking these two points up with the networking subsystem maintainers in Debian, or doing some more research to see if there’s something I’ve missed.
Firewalling
Although we do deploy firewalls at the edge of some of our networks, much of our firewalling strategy (just part of a bigger security strategy, of course) is based on Linux iptables being deployed on each host, with some very specific, tailored policies. In the IPv6 world, the equivalent tool is ip6tables, which is very similar but includes some necessary differences to cope with changes in the ICMP specification and so on.
In order to construct our iptables rules we use some fairly complex Template Toolkit templates as part of a home-grown configuration management system we call rb3, which is able to combine system and service parameters in various ways in order to build configuration files. For iptables rules, where various bits of configuration data need to be merged and handled to build a fairly complex single file, this turns out to be quite useful, especially when it allowed me to start generating both iptables and ip6tables rules from a single template. Although I haven’t replicated all the functionality and service definitions from our iptables setup yet, it wasn’t too hard to get the basic rules to allow us to configure IPv6 and provide port 80/443 services; eventually we will end up with near-complete parity between the two rulesets. This is important, because one of the potential risks of deploying a dual-stacked host or network is introducing inconsistencies between the firewalling policies, which could include unexpectedly permissive rules. Writing both sets of rules from the same set of data and templates helps us minimise those risks.
Testing and deployment
Before changing any IT service, testing is important, but in fact once I’d established that the basic application support was there and that the configuration was all in place (including a test dual-stacked service name) all that was needed for the blogging service was to test that IPv4 only and dual-stacked hosts could use the service okay. Testing from a few different networks (for example at home, if you have IPv6 there) and a few different browsers helped gain confidence in the configuration. For a more business critical application, it’s likely that we’d undergo more extensive testing, engaging a wide range of different users with different networks and platforms to help us, before adding an IPv6 record to a main service interface.
The results
The blogging service has been live with IPv6 for a few months now, and we haven’t had any reports of problems. We have seen 20,000 distinct IPv4 hosts and 120 IPv6 hosts, so around 0.5% of all hosts accessing this service used IPv6. It will be very interesting to see how this figure changes over the course of 2011 (I thought about generating month by month statistics to see whether there was any growth since November, but the numbers are probably too small to be able to draw any conclusions from such an analysis).
IPv6-enabling the blogging service, although it didn’t result in a sudden rush of IPv6 traffic, provided us with some valuable experience not only in the basics of IPv6 service deployment but seeing where things fitted in with our particular style of service deployment and configuration, and I now feel confident that we can deploy additional services as the need arises. It’s likely that we’ll add IPv6 support to some more small-scale services before too long (the GNU/Linux shell service might be a useful one to assist in debugging for others around the University looking to play with IPv6, for example) and to play our small part in World IPv6 day in June, as Guy already mentioned.