This blog post is intended to help ITSS in Oxford better understand how the centrally provided Odin network service (delivered as part of the TONE project) fits together to provide connectivity to their own local networks. It is also hoped it will assist them in assessing the impact of any maintenance work we might need to undertake in the future on the Odin network.
One of the key deliverables of the TONE project and therefore key requirements of Odin, was to provide resilient connectivity to FroDos. Essentially this means connecting the majority of FroDos (options 1 & 2) up to more than one upstream core router to reduce single-points of failure. This also reduces the disruption customers should experience during planned maintenance exercises such as software upgrades on core routers (provided we do one at a time of course). As an example, I note that my colleague Rob Perkins recently undertook this exercise to patch the Odin core routers against a nasty TCAM bug we hit on one of them back in March.
Therefore as Odin was designed to largely eliminate single points of failure, we expect that the visible impact of say, rebooting one core router, should be minimal. In fact, we would hope such work should not be noticed at all by most users. A similar activity on the UBN would have taken far longer and would have required far more time and effort. Both in terms of preparation and communication to inform users of the impending disruption it would cause!
Please note that as you read this blog post, you’ll notice that some indicative link speeds have been provided in some paragraphs below. The author notes that over time, these will likely increase as the needs of the University grow. They are however correct at the time of writing this post.
Several references are made to provisioning ‘options’ in this post for FroDos. For more information on this, please see the Odin SLD here.
The Odin core network consists of four HPE FlexFabric 12910 switches interconnected in a full-mesh. Each link between two core routers consists of four 40Gbps links bundled together in an LACP port-aggregation.
These routers provide resilient connectivity in pairs to the majority of Odin customer FroDos (known as options 1 & 2 in the Odin SLD) using links of 1Gbps, 10Gbps & 40Gbps. I’ll explain the Odin FroDos and the connection pair concept later on in this post.
The naming, numbering & locations of the four core routers in Odin are as follows:
|Radcliffe Observatory Quarter MDX room
|Old Indian Institute MDX room
|Museum Lodge MDX room
|Old Road Campus MDX room
The eagle-eyed amongst you might have noticed the numbering depicted above being binary in nature (or at least decimal representations of binary numbering). This is deliberate and is a core building block of how we number networks and other elements in the internal Odin network infrastructure.
In keeping with the legacy UBN device naming tradition, we prefix each Odin network element type with a letter. The ‘P’ denotes an Odin core or ‘Provider’ router from MPLS terminology which aligns with their function within the Odin network.
Odin Perimeter Network – Naming & Numbering
The Odin perimeter network consists of two HPE FlexNetwork HSR 6804 routers. These provide resilient connectivity between the University and the Internet via the ‘JANET’ network (maintained by JISC).
These routers peer using BGP with the JANET routers in each MDX room (also a JANET PoP) over two 10Gbps links bonded into an LACP port-aggregation at each location. The routers then connect downstream to all four P routers in the Odin core network using one 10G link per P router.
Again we prefix each Odin network element type with a letter. The ‘G’ denotes an Odin perimeter or ‘Gateway’ router.
The naming, numbering & locations of the two perimeter routers in Odin are as follows:
|Primary (active) perimeter
|Radcliffe Observatory Quarter MDX room
|Secondary (standby) perimeter
|Old Indian Institute MDX room
You’ll see that we don’t give the perimeter routers any special binary numbering. However, we have kept the concept of having only one of these routers providing ingress and egress connectivity for the University at any one time. We did consider an active-active routing scheme out to JANET however decided that the cons of such an approach outweighed the pros overall.
Odin FroDos – Connection Pairs
There are currently only four connection pairs into Odin permitted. These are:
- PROQ & PIND = 1+2 = 3 – FroDo numbers beginning 03
- PROQ & PMUS = 1+4 = 5 – FroDo numbers beginning 05
- PIND & PORC = 2+8 = 10 – FroDo numbers beginning 10
- PMUS & PORC = 4+8 = 12 – FroDo numbers beginning 12
In a handful of corner cases, resilient connection from FroDos to more than one P router hasn’t been possible. These provisions are refered to as option 0 in the SLD. In these cases, we do however still nominate a connection pair and reserve the same line card/interface combination on a second P router. This means that we don’t get out of step with our numbering scheme and it also provides an easy upgrade path from option 0 to options 1 or 2 for these sites should the resilient path become viable in the future.
Odin FroDos – Naming & Numbering
On the legacy UBN, the ‘FroDo’ (Front Door) PoP switch installed at each College/department building had a unique number. We opted to stick with this concept on Odin but to make the number longer. The longer number essentially means we can squeeze more useful information into it and hopefully makes it’s meaning clearer at a glance.
Essentially an Odin FroDo number takes the format ‘xxyyzz’ where:
- XX = The connection ‘pair’ number (see above)
- YY = The line card to which the FroDo connects to on each of the P routers*
- ZZ = The interface number on a line card to which the FroDo connects on each of the P routers*
* Note that we keep these the same on each P router when provisioning connections to make this scheme work and also for our own sanity – every little helps!
Odin FroDos are named using their geographic location or address within Oxford wherever possible.
The FroDo switches we deploy are HPE FlexNetwork 5510 series (single-occupancy 2/10Gbps & multiple-occupancy 2G deployments) & FlexFabric 5940 series (single occupancy 40Gbps & multiple-occupancy 10Gbps deployments).
Odin Interaction with UBN
At the time of writing, we have eight 10G links between our four Odin core P routers and two UBN core B routers. These were provided to ease the transition process between the legacy and new networks and are providied on a temporary basis until such time that the handful of customers still connected to the UBN are finally moved on to Odin.
A detailed discussion of why the UBN and these links are still here today is beyond the scope of this post. However the author hopes that the UBN will be fully decommissioned by the end of 2017. If you haven’t read the (rather dated) post about the UBN and it’s naming/numbering conventions, we suggest you take a look here.