syslog

As already referred to in recent blog posts, and thanks to OUCS, some members of Sysdev, myself included, were fortunate enough to attend the UKUUG 2011 Spring Conference. Among the many talks one which particularly stood out for me was a talk given by Simon Wilkinson from the University of Edinburgh about the Git Distributed Version Control System (DVCS). Although this was a 90 minute introductory talk its level was well judged, introducing the fundamental topics that a user currently familiar with a Centralised VCS would need to understand to start using the Git VCS by stepping through a set of basic workflows, while explaining some of the different distributed development file versioning scenarios using directed graphs.

In case this is a bit unfamiliar to you, a VCS is invaluable to developers as it’s where all the shared resources for a project are stored for safekeeping. It records not only source code, configuration, and documentation, but also each and every change in those project resources, so changes can be tracked and reverted if necessary. It’s called ‘Version Control’ because it allows, and promotes, application of your chosen versioning scheme to files. It’s also sometimes called ‘Source Control’, because it is commonly used by software developers to manage changes in their source code. This blog post considers the VCS from the multiuser point-of-view but a VCS can also be invaluable for a single-developer project.

My own experience with Version Control traces a path through RCS, CVS, and Subversion, each of which build on the previous to develop the sense of working with a single shared, centralised repository. In this sense they (particularly CVS and Subversion) are Centralised VCSs. Each user working with a Centralised VCS typically checks out a particular current set of files from the central repository before making changes. After making changes locally the user then accesses the shared repository across the network for VCS interactions such as comparisons, commits, updates, etc. For any activity that needs access to the central repository the user is competing with others for access to the same shared resource. Since the resource is shared it carries with it all the baggage and overheads of locking to prevent leaving the resources in an inconsistent state.

Git has a handy solution to this: when you ‘check out’ a project repository using Git you are actually ‘cloning’ it – you get a backup of the repository, lock, stock and barrel, including all the history. This neatly side-steps some of the contention and locking overheads, achieving some noticeable performance gains since it makes many operations local which would, with Subversion, have been remote repository interactions. This is one way in which Git achieves some significant performance gains but this is not the only difference. There are other features that Git includes to make VCS users happier that seem to have been missing from Centralised VCSs, such as squashed commits, simpler and less resource-heavy branching, and simpler repository file formats. Yet another benefit that Git brings to the table is that equivalent repositories are much smaller than other VCSs.

Sysdev currently run a Subversion VCS service for use by OUCS staff and OUCS-run projects. We also count ourselves among the users of this service, since it is used by our home-grown configuration management system rb3, as well as for the projects undertaken by Sysdev staff. Subversion was originally designed as a largely compatible replacement for its predecessor, CVS, and as such, it shares much in common with CVS syntax. In turn, there is some familiarity to Subversion users in the syntax that the Git developers have chosen for the Git toolset. Tools are available to migrate repositories from Subversion to Git (git-svn), just as tools were developed for migration from CVS to Subversion (cvs2svn). However, apparently Git’s design goals were not primarily to be a direct Subversion replacement, and when you are used to a Centralised VCS using a DVCS can take a bit of getting used to. I’m no expert Git user, but as a long-standing user of other VCSs I believe that Git has speed and efficiency improvements, along with other convenience benefits, that are well worth consideration by our users. Naturally, for a team such as ours to consider providing a DVCS service such as Git the demand for it would have to be expressed by our users, and assessed.

Git is an example of a project that was born from Linux kernel developers wanting a better tool, and as such it fits well in a UKUUG Conference, but at its core Git has much wider appeal to developers in general. This seems to me to be typical of the UKUUG Conference. If you’re not part of a UNIX team you may think that a conference organised by a UNIX users group is too specialised to be relevant, and perhaps not really your thing. If you think along those lines I’d like to suggest some reconsideration as these are useful events, with talks likely to be relevant and of interest to a broad range of IT professionals. As if to highlight that a recent announcement was made: the UKUUG have changed their name to FLOSS UK, which stands for Free and Libre Open Source Software, the name change reflecting a broader scope and a correspondingly wider appeal.

At OUCS, the 6-monthly release of Ubuntu Linux has become a source of regular attention from Sysdev and the networks team, because we provide a public mirror of Ubuntu, and several other Free software project repositories. Owing to the relatively sophisticated download distribution system used by Ubuntu, and the fact that we are registered as a mirror with a relatively high capacity (1Gb/s uplink) the release day will generally find us keeping an eye on server load, traffic graphs, etc. In the olden days our mirror was behind a 100Mb/s firewalll; once that was removed, we needed to tune the Apache configuration of the mirror service to cope with the number of simultaneous HTTP clients generated by the release, and so on.

The graph above shows the affect the mirror traffic had on our gigabit uplink port to the rest of the OUCS data centre network. As you can see, the traffic came close to maxing out the capacity of the link, and for the first time, we observed noticeable service degradation of other services sharing that uplink. You can see the effect on Weblearn at the Think Broadband site. We still don’t know quite what changed to push things over the edge, but I suspect that a combination of factors including many outside our control.

Lovely though Ubuntu is, it was fairly clear that it should not be swamping traffic at the expenses of critical University services such as Weblearn, so we worked immediately to resolve this. A short-term configuration change of returning 403s for half the mirror traffic was put in place, followed by some investigation of readily available rate limiting software for Apache. We looked at and deployed mod_cband, but this turned out to be relatively inflexible and quite difficult to get well-tuned configurations for, meaning that over the weekend after release downloads were unnecessarily slow.

In the calm of the following week, I looked at alternative approaches and turned to the invaluable LARTC HOWTO, dusting up on some concepts I’d been vaguely familiar with but never used seriously (beyond using wondershaper to improve the interactivity of my home broadband connection).

There are two important pieces of advice here; firstly, it’s well worth investing a couple of hours digesting the whole traffic shaping chapter of LARTC; it’s really well written but covers some complex topics. Secondly, as it points out, CBQ, the queueing discipline you’ve probably heard of, might not be the obvious choice you thought it was.

I spent a fun afternoon playing with various different configurations as I went along, simulating a heavy load (of course by this time the surge in demand for Ubuntu had receded) from my desktop in the office, and linux.ox.ac.uk (on the same local network as the mirror service). I concluded that CBQ had far too many parameters which you really had to have a deep understanding of to use successfully, and as a result I didn’t arrive at a configuration I was happy with after half an hour of testing. By comparison, after about 10 minutes of experimention, I arrived with the HTB based configuration we now have, which limits outgoing mirror traffic on each server to 350Mb/s (two servers share the same 1 Gb/s uplink; 300Mb/s ought to be enough for other services) without unnecessarily limiting individual clients. In addition I added some SFQ fairness, which is worth having on the mirror service regardless of the need to rate-limit. It should go without saying that the rules is in no way a general recipe, and almost certainly has the potential for improvement, but I include them for interest:


# Deletes previous configuration
tc qdisc del dev $DEV root
# Adds a new htb qdisc at the root
tc qdisc add dev $DEV root handle 1: htb
# Adds a new class, 1:1 to the root qdisc
# - rate-limits
# - burst set with trial and error (probably higher than needed).
# Needs to scale with rate
tc class add dev $DEV parent 1: classid 1:1 htb rate 350mbit burst 1m
# Adds a new qdisc to the previously defined class to add fairness
# - perturb factor is as recommended in manpage
tc qdisc add dev $DEV parent 1:1 handle 10: sfq perturb 10
# Sends outbound mirror service traffic to the rate-limiting class, 1:1
# - priority should not be '1' since this will affect other
# traffic.
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match \
ip src $MIRROR_IP flowid 1:1


Further work is planned to prioritise traffic going to University IP addresses over external ones.

Of course the acid test won’t be until October, when Ubuntu 11.10 is released…

With thanks to pod and other Sysdev team members who worked to firefight this incident, the networks and security team for providing the pretty graphs, and for the Weblearn team for providing valuable assistance during the incident.

We are very pleased to announce that OxFile, our new service to support the exchange of large files, is now available for general use.

https://oxfile.ox.ac.uk/

OxFile is a web service that supports the exchange of large files with people inside and outside of Oxford University, and is fast and simple to use. It has been designed to assist users who need to exchange files that are too large to send via the University’s Nexus email service, too large to upload to WebLearn or SharePoint, or that need to be exchanged with people who do not have access to WebLearn or SharePoint.

Example Use Cases

Some of the use cases that OxFile was designed to support include:

• Sending print-ready documents such as Alumni magazines and academic articles to press / publishing houses
• Receiving academic data sets from collaboration partners and commercial suppliers
• Thesis submission
• Upload of MP3/MP4/AVI files such as recordings of lectures for inclusion in a web site
• Sending large files for technical assistance (e.g. RAR files that cannot be decompressed, or system logs for analysis)

Developed in Response to User Suggestions

Development of this service was triggered by the Innovation Team at OUCS. The Innovation Team receives and responds to service suggestions sent to OUCS, and in this case recommended that a project be established to commission a new service to meet reported user needs. It has been a long while since the original approval was given, during which time a lot of software development and user testing has been carried out.

Our goal has been to provide a service that is as easy to use as the many commercial and free alternatives, but which meets the unique requirements of Oxford users. We are grateful to everyone who helped test this service – the fact that the first round of user testing resulted in a major reworking of the user interface is testament to the importance of your input to the development.

La-arge Files

How big is a “large” file? OxFile will accept individual files upto 20GB, and when exchanging multiple files together you can have upto 40GB in a bundle. Many users exchange data that would fit on a DVD, but which is too large for most other immediate / free options (including Dropbox which has a limit of 2GB for the free offering).

These limits have been set primarily to reduce the impact of inadvertent errors on other users, and will be reviewed periodically.

At this stage it is hard to predict with much confidence how heavily this service will be used. Accordingly we have given it a substantial initial transfer capacity (290GB), but will monitor this closely during this early life of the service and make  adjustments to accommodate greater volumes if required and possible.

Complementing Other Services

As mentioned above, OxFile provides functionality similar to that available from a number of existing OUCS services. We have designed OxFile to complement, rather than compete with, these other services. At the time of writing, WebLearn has a per-file upload limit of 60MB (uploading via the web interface; WebDAV accepts files upto the site limit, typically 1GB), Nexus email and the Oxford message submission service have message size limits of 100MB, and SharePoint has a web upload limit (thought to be about 80MB) and a default site limit of 5GB.

Of course, OxFile can only be used to hold files for a few days, and does not offer the rich process-specific functionality available in these other services.

Further Information

If you wish to read more about the service then the following pages may be of interest:

• OxFile service description
• OxFile service level description
• OxFile service help page

Any other enquiries about the service should be sent to help@oucs.ox.ac.uk in the first instance.

Along with pod (who posted his introduction on the topic on this blog recently) and some others I also went along to the UKUUG conference in March. As we’ve come to expect it was a well-organised event, and this time it was held in a very nice location which made it all the more enjoyable; although we did suffer occasionally from oversubscribed talks with people sitting in the aisles!

One strong theme in the conference this year, which I will explore in this blog entry, was DNSSEC. There were three separate talks by Jan-Piet Mens, starting with a general introduction to the technology for those familiar with DNS, followed by a pair of talks discussing techniques and tools for signing and validating names with DNSSEC. I won’t repeat any of the content of the talks here; there are plenty of general resources about DNSSEC out there, and you can get the slides for these talks from the UKUUG web site:

• Introduction
• Validating
• Signing

This topic has been of interest to me for some years, and I recently implemented full DNSSEC validation on the team’s standard server configuration (based on Debian ’Squeeze’). However as with all technologies, it’s good to be reminded of the basic principles in technologies you work with. This serves to challenge any doubtful assumptions or misunderstandings you may have (unwittingly) absorbed, as well as gain fresh insights on how to solve some of the more tricky problems. I was, therefore, pleased to see that Jan-Piet’s presentation on validation focused on unbound, which I’ve used as a validating resolver on our Debian systems.

Probably the biggest open question in my mind as a result of that talk was how we should expect end-user machines (be they desktops, laptops or mobile devices) to have access to a fully end-to-end secure DNS channel at all times. For example, if I recall correctly, the venue’s wireless facility offered a DNS server which did not pass through the required parts of the DNS response to allow a local validating resolver to function; and if (as many access networks do) external DNS connections had been blocked, there would have been no way to receive secured DNS responses at all, without recourse to a VPN. There are clearly some interesting autoconfiguration challenges here, and I found myself musing over how I’d like my laptop to behave when faced with these various restricted networks. Perhaps there is an enhancement request for Network Manager to condense out of those thoughts.

The other part of this problem lies in default OS configuration; at the moment, the channel between the traditional stub DNS resolvers and their nominated recursive resolvers is unsecured, and represents a significant attack surface (depending, of course, on the local network topology). One answer would simply be to install DNSSEC validating resolvers on all hosts, but this represents a big change compared to the tiny, unmanaged stub resolvers which exist on nearly every client platform to date. It will be interesting to see what becomes recommended practice here; whether there are other plausible alternatives to securing that channel; and which OS vendor will be first to make a local validating resolver part of their default configuration, given the above concerns (and rapidly evolving landscape of DNSSEC configurations in the wild, and software to support this: an oft-repeated point in the talks was that one should always be using the latest and greatest DNSSEC tools because of the rate of change in this area).

Further up the stack, towards user applications, there are clearly lots of opportunities for application-level DNSSEC validation, and the Firefox DNSSEC Validator extension is just one.

The signing talk was also of great interest; among many other useful pieces of advice, I learnt that BIND’s built-in tools appear to have become much more user-friendly and automatic than the last time I looked; however I have yet to have a chance to sit down and start signing any zones of my own (DNS is managed by a separate team, so there will be less scope for experimentation with signing here at OUCS).

Coincidentally, during this conference, the Comodo CA rogue certificates incident came to light, and this provided a nice reminder of how critical security mechanisms such as SSL and DNSSEC are becoming, and the importance of limiting exposure to incidents of this sort. Ironically, one of the rogue certificates issued was for addons.mozilla.org, which distributes the above Firefox extension!

This tied in quite nicely with some remarks from the talks about other opportunities DNSSEC provides, over and above the traditional names and numbers: now we are beginning to develop a global, distributed and verifiable database, publishing trust anchors such as SSH and X.509 certificate fingerprints in the DNS becomes realistic. SSHFP has actually existed as a DNS resource type for some time, but perhaps now becomes much more useful.

DANE is an standardisation effort to specify a way of doing the same with X.509 certificates and TLS. This is an interesting political, as well as technical point; some may feel that with such technologies on the horizon, the reign of the traditional certificate authorities is drawing to a close, and that systems like DANE can actually provide greater assurances than we’ve come to expect from commercial CA providers (and the problems distributing CA certificates for smaller, organisation-scoped CAs). The conspiracy theorists may claim that DNSSEC with DANE is no better than the small number of commercial X.509 CAs, and that both could easily be subverted by government or other powers, but DNSSEC has one powerful ability which counters this argument at least to some extent; anyone who is validating names with DNSSEC is at liberty to insert trust anchors relating to any part of the DNS tree they choose (so, for example, Sysdev obviously has a significant interest in the ox.ac.uk DNS subtree; we might therefore configure our resolvers with the DS records of ox.ac.uk if they existed and we could retrieve them securely — in our case, by popping round to the next office).

By way of closing, I note that this year saw a new academic workshop on DNS security: SATIN. I haven’t yet had a chance to explore the papers on this site, but there are several topics of possible interest there. It seems like interest in this area will continue to increase over the next couple of years.