Network Development Team

Our targets for the IPv6 deployment for this week have been mostly successful

• Enabling IPv6 to first half and now the second half of the OUCS server room has been completed, we’ve had no complaints of disruption from other teams so this appears successful.
• All our development hosts are now IPv6 enabled. These were the first hosts enabled which was useful for setting up our configuration management systems handling of IPv6.
• Two of our four IPv4 stratum 3 NTP servers are now IPv6 enabled, providing a live service currently at ntp6.oucs.ox.ac.uk. In months to come I suspect we’ll add AAAA records for ntp.oucs.ox.ac.uk however for now we’re being cautious.
• At IT support staff request the IRC service at irc.ox.ac.uk has been IPv6 enabled ahead of schedule. This was more rushed than I would have liked but we were able to test on a development host and the change was completed with the service live without downtime or disconnections. Sadly the jabber/xmpp service which is present on the same physical server received less testing and wasn’t successfully IPv6 enabled, AAAA records for this service have been removed until time can be set aside for proper testing and deployment. For this service (which is mainly used by IT Support Staff) we decided to add AAAA records for the normal service address (rather than use, say, irc6.ox.ac.uk) but haven’t seen any reported issues yet.
• Our teams main database server has been IPv6 enabled, this is not a direct service to IT support staff but provides information to many of the tools and backend processes.

The following were originally planned for this morning but are behind schedule:

• IPv6 enabling the webcache service – this simply hasn’t had time set to it due to other demands. It’s reasonably straightforward (a little Squid configuration, some ip6tables work, some testing) but I don’t believe the migration should be done live like the irc.ox.ac.uk service was and although little used nowadays it has a larger number of users than the irc service.
• Outside the IPv6 project the migration of one of the university main DNS servers to new hardware will be delayed. It was ready on time this morning but I’d like to do more testing before deploying, I wasn’t comfortable that it had been tested enough.

So my work for the next four days is

1. Testing the replacement DNS server with the aim to deploying in next weeks Tuesday slot instead (21st September)
2. Planning/preparation for IPv6 enabling the webcache in the same slot
3. IPv6 enabling the host that provides our team website IT support staff use for internal network facilities, this is also a IPv4 NTP stratum 3 host so we can add it to the ntp6.oucs.ox.ac.uk service once enabled.
4. Ahead of schedule, adding the network monitoring system host itself to IPv6 as it’s also a IPv4 stratum 3 NTP host, but probably not doing any work on monitoring via IPv6 until the planned date due to time.

We’ve a timeline of work for the IPv6 deployment for the next month which might be of interest to some and at the same time we’ve work updating the hardware and operating systems for certain core services. There’s other services that will also need to support IPv6 but I don’t want to plan at an operational level for much more than a month ahead since unexpected additional tasks tend to crop up and skew the dates.

In brief we’re enabling IPv6 to our server room and starting the first core service upgrades, in some cases mixing the migration in with a standard hardware/software refresh (where combining the tasks makes the migration simpler rather than harder). The dates reflect the weekly JANET at risk period for maintenance, Tuesday 7:00am-9:00am.

You might think that we could enable core services faster however the above is one persons work from a four man team – we will have normal duties and other projects underway during this period (the team manages 22 public services and about 13 internal services (physical network monitoring, telecoms integration etc).

The enabling of the first part of the OUCS server room this morning (and the background research/testing as well as the notes this section is based on) was done by another team member however the essential features are that we’re providing IPv6 to the server room with router advertisements disabled, meaning dual stacked servers run by other teams won’t suddenly obtain an IPv6 address automatically. We can get on and IPv6 enable our teams services with a statically configured address without disrupting others. The configuration on a cisco device looks like

ipv6 nd ra suppress
ipv6 nd prefix default 2592000 604800 no-autoconfig

The Cisco IPv6 command reference for nd is available online. In the example above ignore the numbers which are just defaults for valid-lifetime and preferred-lifetime in seconds (normally for a specific network prefix, which we do not need in this case) via router advertisement. The important part is the ‘suppress’ and ‘no-autoconfig’.

The first line tells the router that for ipv6 neighbour discovery it should suppress periodic router advertisements. The second line instructs the router that if asked it should tell the client not to use autoconfiguration for the network. In technical terms, when it gets a request from a client it will respond but in the Prefix Option (see RFC2461) the A-bit is cleared, meaning that the client must not use autoconfiguration for that prefix.

For further information see this c-nsp thread on RA/RS messages and also this other thread.

We’ve made some progress, we’ve reached agreement with the security team that we can enable IPv6 on the OUCS machine room hardware starting 7th September, which means we will be able to start making our core network services IPv6 capable. We’ve already had the political approval for this so this week we’re testing before the change.

I’m currently working on a hardware refresh of the core DNS/DHCP service so there’s a good chance that when the new DNS servers come online they will be able to handle queries over IPv6. The other core network services might not take long to follow as we’ve already done a lot of the background work needed. By the time this is done we can start deploying IPv6 to units that are interested in early testing.

I did an IPv6 talk today for roughly 40-50 of those not able to make the recent talk at the IT support staff conference. I gave a handout to anyone interested in taking part in early testing, a copy of which is below. It’s important to remember that the university is made up of roughly 200 politically independent units, most of which have their own local IT support staff. There’s a lot of variety in skill sets/specialities and this is compounded when either managers or IT staff could be the audience so it’s sometimes tricky to get the technical level right in the handout.

So the following are steps we are asking units to have taken before taking part in the IPv6 early adopters/testers.

1. Are you running Spanning Tree?

Please deploy Spanning Tree Protocol (STP) on your switches. This prevents network loops in your network, symptoms of which will be any of

• excessive broadcast traffic

• unexpected traffic arriving at interfaces

• lower than expected network throughput

This is important because network loops are a fundamental issue in a complex network which STP resolves. Your switch vendor support channels may refuse to assist you with other issues you raise while looping is evident on your network.

Rapid Spanning Tree Protocol (RSTP) can be viewed as a newer implementation of STP, this is also fine and will give a much shorter ‘repair’ time (convergence) to repatching if your switches support it.

This isn’t our only switching advice but it’s the bare minimum you should have for a sane network.

Links:

http://packetlife.net/blog/2009/oct/15/stp-your-friend/

2. Do you have a helpdesk ticketing system?

You should be able to track a users open issues, track new issues, see what work has been done on troubleshooting an issue and who is dealing with it. If there is some dispute or re-occurrence and a user queries what happened to their previous issue, you can then look it up and see what the last correspondence was and why it was resolved.

If you don’t have one of these the ‘RT’ system is recommended and widely used.

Links:

http://bestpractical.com/rt/

[edit] I should have noted in the original handout that NSMS can offer RT as a hosted service to units, they have price details online.

3. Do you have a network diagram?

You should have or be able to produce a physical and logical network diagram for your network. It doesn’t have to have every workstation you own but you should be able to diagram your core infrastructure.

This will save you time when planning network changes and IPv6 deployment. It may save expensive (man-hours or equipment) design mistakes.

Ask another team member to independently check the diagram or separately produce their own to test it is correct.

4. Do you have a central documentation system for the IT team?

Can your IT staff record information about a service, server or similar that other staff can then look up? It could be a wiki or a content management system like Plone or even a shared folder full of MS Word documents (not a recommendation but better than nothing), as long as it’s actually usable. Does your team find it easy to use, are they using it?

This is important because it reduces duplication of effort. Odd corner cases and workarounds are troubleshooted/discovered and documented once, from then on staff need only implement the solution that was previously found, no matter if the staff member is ill, has left or worse.

If server/service X goes down then the documentation should let junior members of staff bring it back up.

5. Are your hosts time synchronised?

You can use the NTP servers that make up ntp.ox.ac.uk and ntp.oucs.ox.ac.uk. These means that when you are troubleshooting you can know for sure that the time period from logs on one server match that on another. At least have your servers and key network devices (firewalls etc) synchronised with a working NTP service.

6. Are you on top of your IT in general?

1) Check your unit doesn’t have unhandled security blocks on hosts pre-dating 2010

[internal only link, sorry] https://networks.oucs.ox.ac.uk/webauth/blocks

If it does, chase each one up with security@oucs.ox.ac.uk , investigate and clear them up.

2) Check your units university firewall webserver exemption details are correct and don’t include dead hosts or unexpected internal webservers, unexpected items of critical network infrastructure that happen to have a web interface and similar.

[internal only link] https://networks.oucs.ox.ac.uk/webauth/firewall?showog=http_servers

To change entries email networks@oucs.ox.ac.uk

3) Look at your internal helpdesk queue, are you coping? If you can’t tell what is going on, or there’s more than perhaps 30 open tickets per supporting staff, then it might be time for some form of intervention.

The aim of the above wasn’t to put up barriers to adoption but to ensure that a basic level of network sanity was present in those networks taking part. I could have made the list longer (ideas welcome in the comments) however some of the above are quite large projects to undertake for a small unit currently without and facing a 10% budget cut plus employment freeze. Aside from IPv6 perhaps our team could produce a best practise recommendation checklist for units that they can self audit themselves against (and keep the results to themselves) if they wish.

At the ICT conference I gave a talk alongside Bob Franklin from Cambridge. Bob covered IPv6 in general and then I covered a deployment case study using our teams deployment plans but trying to make it as general as possible so that I’m describing

• a connection to the internet
• having services you run
• having customers you serve

…which should hopefully make it applicable to Oxford or Cambridge and any subunit.

This is a rough text version of the talk, I didn’t speak from a script so it may differ slightly and in the talk I did skip a few minor ideas that Bob and I had accidentally overlapped on.

Note that you could make a start on these steps today, you do not need a IPv6 connection as your first step. A working connection is actually quite a late step in the preparation.

1) Perform a network device audit

In roughly 2008 we started an audit of all switch, router and firewall hardware on the backbone to record hardware and software versions.

The resulting list was then compared to what the vendor said the device could do. Some devices might have no IPv6 support, others might be partial, others might support it but purely in software as opposed to hardware. For a firewall supporting IPv6 in software might mean that the device can deal with multi Gb/sec rates of IPv4 traffic but only 80Mb/sec of IPv6 which is not workable.

When looking to replace items you may decide some devices might not need IPv6 support in their lifetime – it might be operating as a simple switch with no access controls nor management via IPv6.

Be prepared to test a sample device with the software version you are planning to run – as fairly new implementations IPv6 commands may have defects or simply fail to run. You might need to upgrade your device vendors operating system on the device or raise a support case with them. It’s usual to upgrade the software for security issues however it’s typical to stay on a release that it known to be stable so you may uncover the need to upgrade as you test your intended IPv6 deployment.

Rather than spending money to replace a large quantity of equipment, plan to replace devices during normal hardware maintenance cycles. That is, if a non IPv6 capable device is up for replacement in a year, then perhaps you can wait until this time to purchase new hardware that is ensure to support IPv6. Ask the vendor for a sample before you purchase many units and test it. Complain with technical detail if you find flaws while attempting your planned IPv6 commands.

For other equipment you may decide it’s necessary to make a short term purchase in order to keep your deployment plans moving.

2) Perform an audit of services

The public services offered by our team were fairly easy to enumerate (DNS, NTP, SMTP etc), for each we recorded the software version and research the IPv6 capability. Where it was lacking the expected native IPv6 release version/date was recorded.

Slightly more tricky is enumerating the tools we offer that provide a service to our customers (in our case IT support staff), such as web interface that take an IPv4 address as the query in order to show DHCP reservations, or anywhere an IPv4 address is entered to register a device. In each case the tool needs to be rewritten to support IPv6 (which might be minor or major code changes depending on the tool) or needs to be planned to be replaced.

Then lastly behind the scenes we have all the scripts that ‘glue’ together the services; cron jobs that retrieve database data, scripts that update service configuration, backup and restoration scripts. We have roughly 150 of these, and an audit needs to be done to find places for instance with regular expressions looking for IPv4 addresses, replacing them with a regular expression library of some sort that can match either IPv4 or IPv6. I’d suggest using a regular expression library that can do this is better than having to write two separate scripts, or a script full of twice the logic decisions.

3) Build an IPv6 test network

We built a IPv6 only test network and deployed the same production services we run on IPv4 but entirely in the IPv6 test network

We documented any differences in configuration syntax and behaviour for setting up the service under IPv6

We also documented minor aspects, changes in utility names (e.g. ping6 instead of ping, ip6tables instead of iptables) since this step is also important in getting staff experienced in the configuration under IPv6 and also root out any surprises

Note that you do not need a functional IPv6 connection to any external network to undertake these steps.

4) Write a formal deployment plan

Can be peer reviewed, helps to straighten out all the minor aspects of deployment that might not have been considered.

The plan can then be shown to management and other teams to make them aware of the situation

We then approached a unit (e.g. customers) we knew to be technically able and constructive to work with (for a department this equivalent might be a research group or similar subset of users), asking if they would like earlier than normal access to an IPv6 connection supplied to their unit in return for feedback from them and under the understanding that this was not a production service – there might be some unforeseen consequences during development.

Having had a good experience with the first, we approached two other customers but I had made a mistake in our planning. We’d considered all the technical aspects of deployment but not the political aspects. The questions included:

• Can you prove that the university isn’t about to demand our IPv4 space back?
• Can you prove when the last IPv4 space will be given out in the university?
• Is the University making any money available to the units for the change?
• The supplied IPv6 connection has to be a fully production service with no issues

By this point we’d also joined the Cisco IPv6 deployment council so that we could give feedback on what IPv6 feature development we wanted from our current switch vendor. It’s under a Non Disclosure Agreement so I’ll not go further into this other than to say it’s technically valuable to us.

5) Produce a formal IP addressing Policy

It took longer than expected to produce the addressing policy, the main concerns being of making mistakes at this point in time that would be impossible to fix without severe disruption later and hence inflicted upon generations to come.

We used advice from technical sources including Southampton and Loughborough Universities as well as our switch vendor.

6) Where are we now?

• We’ve deployed a dedicated server to handle IPv6 traffic in and out of the university in place of passing it through the main university firewall, the later being capable of IPv6 in software only. When the next maintenance purchasing cycle is due this will be replaced with dedicated hardware.

• We’re aiming to supply IPv6 connectivity to our machine room in a controlled fashion (to avoid unexpected impact on other teams), once done we can start IPv6 enabling the core network services.

• The first unit will be supplied with an IPv6 connection as part of the initial trial once our security team is content that they are ready to handle dealing with security incidents on IPv6 based hosts (we’re expecting this to be probably within a couple of months).

• Our current university DNS/DHCP interface for IT support staff cannot handle IPv6 but the implementation of the replacement in the Oxford environment is a little complex. The replacement was hoped to be ready for mid August but this now looks impossible (this is a large project so I’ll do a separate post to cover this).