Sorry it’s been a little while since our last post. We’re just coming to the end of the summer vacation, the point when we (and most other IT support staff around the university) get things ship-shape in time for the start of the new academic year. It’s been hectic, to say the least, with a number of exciting projects to talk about in these pages.
Several weeks ago now we upgraded the university’s connection to JANET to 10 Gigabits per second (from 2 Gbps). JANET-UK is the ISP for UK universities and in this region of the country we form part of something called the Thames Valley Network (TVN), which connects institutions to the JANET network, and each other. Here I’ve drawn a simplified image of the current TVN topology:
Thames Valley Network
It was back in 2007 that we upgraded from 1 Gbps to 2 Gbps. That was a very exciting project, delivering some much needed resilience improvements by having two 1 gbps feeds on separate line cards in our edge router. These upgrades had been driven by our discussions with the university’s auditors, so next time you have the auditors come knocking, remember there can be a silver lining to that cloud! At the time we also were in need of the bandwidth upgrade, as Internet traffic continued on its ever upwards climb. Here’s a rather cute graph using data from the OUCS Annual Report showing this growth:
There are likely a number of causes of this rise in traffic in the past three years. Services on the Internet such as YouTube, iTunes, and more recently iPlayer, have played a part. Also, our friends in the Particle Physics (GridPP) community are making use of any and all available bandwidth to transfer experiment data around the globe. It was them that in part prompted JANET-UK and ourselves to move for the 10 Gbps upgrade this summer, knowing that CERN ‘s LHC is due to come back on-line.
Under normal circumstances we’d like to keep plenty of headroom on the JANET link, but from time to time these experiments result in about 400 Mbps of extra traffic. Whilst the “2 Gbps” link can take this, that is actually two separate 1 Gbps links using a simple hashing algorithm to load balance. If 400 Mbps is destined for one host elsewhere on the Internet it will hash onto one link and potentially fill it up even with spare capacity on the other link. This is one reason why JANET-UK themselves run 40 and 100 Gbps links in the core rather than bonding together multiple 10 Gpbs links with hash based load balancing. Here’s another graph from the depths of summer 2009, showing one such GridPP session receiving a few hundred Megabits:
A few weeks from now when term is underway, that 1.3 Gbps peak will be a daily average with the GridPP traffic on top. So for us the next step was to move from 2 Gbps to 10 Gpbs, but of course that means a pair of them to maintain the resilience we installed back in 2007. With good foresight we’d planned for this since before the 2 Gpbs upgrade, so all the 10 Gbps line cards were in place in our routers. Not so straightforward for the JANET-UK engineers though, who had to make a few visits to OUCS to upgrade their routers at our site. It was quite interesting for us to get an insight into their operations and the hardware platforms they’re moving to for the Thames Valley Network.
There still remain areas where we can improve on the resilience. Most significantly the new OUCS data centre currently under construction (and due 2011/2012) will likely house a second point of access for the university into the Thames Valley Network (and hence JANET and the Internet). Between now and then we will be working with JANET-UK in 2010 to make some improvements to the fibre routes in the city of Oxford to provide for greater resilience. As you might tell from the above, the work never ends. There will continue to be a need for more bandwidth and resilience, and the university is fortunate to have some great engineers at JANET-UK to help us deliver on that.