OWL Visitor refresh – Timetable

This blog post is an expansion on the previous post about Tawny OWL to give timescales.

The upgrade of the FroDos to re-route OWL traffic from the current system to the new one (called Tawny) is roughly a two stage process:

1) Configure the Frodo to set up the relevant VLANs and L3VPNs.
2) Associate the existing OWL ports with the new VLAN.

Step one can happen without any effect on existing traffic and we will be doing that on all FroDos before we move onto step 2.

Step 2 will be done in batches to aid us in testing successful migration testing. Also, note that if you are a COWLS customer, we will be applying the change to your FroDos just for completeness. Nothing will change with respect to existing connectivity.

The listing goes as follows:

First batch

To be scheduled for morning 3rd March 2020

  • john-radcliffe-3.frodo.ox.ac.uk
  • molecular-medicine.frodo.ox.ac.uk
  • big-data-institute.frodo.ox.ac.uk
  • orcrb-2.frodo.ox.ac.uk
  • wellcome-trust.frodo.ox.ac.uk
  • richard-doll.frodo.ox.ac.uk

Second batch

To be scheduled for morning 4th March 2020 modulo anything arising after first batch migrations.

  • 101-banbury-road.frodo.ox.ac.uk
  • 105-banbury-road.frodo.ox.ac.uk
  • 105-woodstock-road.frodo.ox.ac.uk
  • 106-woodstock-road.frodo.ox.ac.uk
  • 10-parks-road.frodo.ox.ac.uk
  • 11-bevington-road.frodo.ox.ac.uk
  • 11-manor-place.frodo.ox.ac.uk
  • 11-norham-gardens.frodo.ox.ac.uk
  • 11-st-johns-street.frodo.ox.ac.uk
  • 128-bullingdon-road.frodo.ox.ac.uk
  • 12-bevington-road.frodo.ox.ac.uk
  • 12-parks-road.frodo.ox.ac.uk
  • 12-wellington-square.frodo.ox.ac.uk
  • 132-walton-street.frodo.ox.ac.uk
  • 139-walton-street.frodo.ox.ac.uk
  • 13-bevington-road.frodo.ox.ac.uk
  • 13-bradmore-road.frodo.ox.ac.uk
  • 14-wellington-square.frodo.ox.ac.uk
  • 16-wellington-square.frodo.ox.ac.uk
  • 17-norham-gardens.frodo.ox.ac.uk
  • 189-banbury-road.frodo.ox.ac.uk
  • 191-iffley-road.frodo.ox.ac.uk
  • 194-divinity-road.frodo.ox.ac.uk
  • 1-keble-road.frodo.ox.ac.uk
  • 1-museum-road.frodo.ox.ac.uk
  • 21-banbury-road.frodo.ox.ac.uk
  • 239-iffley-road.frodo.ox.ac.uk
  • 23-banbury-road.frodo.ox.ac.uk
  • 25-staverton-road.frodo.ox.ac.uk
  • 25-wellington-square.frodo.ox.ac.uk
  • 2-bradmore-road.frodo.ox.ac.uk
  • 2-museum-road.frodo.ox.ac.uk
  • 32a-little-clarendon-street.frodo.ox.ac.uk
  • 32-jack-straws-lane.frodo.ox.ac.uk
  • 32-st-giles.frodo.ox.ac.uk
  • 32-wellington-square.frodo.ox.ac.uk
  • 33-stanley-road.frodo.ox.ac.uk
  • 33-st-margarets-road.frodo.ox.ac.uk
  • 34a-st-giles.frodo.ox.ac.uk
  • 36-beaumont-street.frodo.ox.ac.uk
  • 38-woodstock-road.frodo.ox.ac.uk
  • 39a-st-giles.frodo.ox.ac.uk
  • 39-iffley-road.frodo.ox.ac.uk
  • 3-hythe-bridge-street.frodo.ox.ac.uk
  • 3-st-johns-street.frodo.ox.ac.uk
  • 3-worcester-street.frodo.ox.ac.uk
  • 41-st-giles.frodo.ox.ac.uk
  • 42-park-end-street.frodo.ox.ac.uk
  • 43-banbury-road.frodo.ox.ac.uk
  • 44-st-giles.frodo.ox.ac.uk
  • 51-banbury-road.frodo.ox.ac.uk
  • 5510-netdev-test-roq.frodo.ox.ac.uk
  • 56-banbury-road.frodo.ox.ac.uk
  • 58a-banbury-road.frodo.ox.ac.uk
  • 58-banbury-road.frodo.ox.ac.uk
  • 59-banbury-road.frodo.ox.ac.uk
  • 5-bradmore-road.frodo.ox.ac.uk
  • 5-broad-street.frodo.ox.ac.uk
  • 5-st-margarets-road.frodo.ox.ac.uk
  • 61-banbury-road.frodo.ox.ac.uk
  • 64-banbury-road.frodo.ox.ac.uk
  • 66-st-giles.frodo.ox.ac.uk
  • 68-banbury-road.frodo.ox.ac.uk
  • 74-high-street.frodo.ox.ac.uk
  • 75-iffley-road.frodo.ox.ac.uk
  • 7-holywell-street.frodo.ox.ac.uk
  • 92-woodstock-road.frodo.ox.ac.uk
  • 99-banbury-road.frodo.ox.ac.uk
  • 9-parks-road.frodo.ox.ac.uk
  • ablethorpe.frodo.ox.ac.uk
  • alan-bullock-close.frodo.ox.ac.uk
  • andrew-wiles.frodo.ox.ac.uk
  • anna-watts-building.frodo.ox.ac.uk
  • aopp.frodo.ox.ac.uk
  • ashmolean.frodo.ox.ac.uk
  • balliol-college.frodo.ox.ac.uk
  • beecroft.frodo.ox.ac.uk
  • blackfriars-hall.frodo.ox.ac.uk
  • blavatnik.frodo.ox.ac.uk
  • botanic-garden.frodo.ox.ac.uk
  • brasenose-college.frodo.ox.ac.uk
  • bsf-swindon.frodo.ox.ac.uk
  • campion-hall.frodo.ox.ac.uk
  • cardo-building.frodo.ox.ac.uk
  • castle-mill-2.frodo.ox.ac.uk
  • catholic-chaplaincy.frodo.ox.ac.uk
  • cavalier-court.frodo.ox.ac.uk
  • ccvtm.frodo.ox.ac.uk
  • chemistry-research-lab.frodo.ox.ac.uk
  • chemistry-teaching-lab.frodo.ox.ac.uk
  • christchurch-college.frodo.ox.ac.uk
  • christchurch-sports-ground.frodo.ox.ac.uk
  • clarendon-laboratory.frodo.ox.ac.uk
  • club-research-lab.frodo.ox.ac.uk
  • club-teaching-lab.frodo.ox.ac.uk
  • cohen-quadrangle.frodo.ox.ac.uk
  • corpus-christi-college.frodo.ox.ac.uk
  • court-place.frodo.ox.ac.uk
  • cripley-road.frodo.ox.ac.uk
  • denys-wilkinson.frodo.ox.ac.uk
  • direct-labour.frodo.ox.ac.uk
  • dorothy-wadham.frodo.ox.ac.uk
  • dunn-school.frodo.ox.ac.uk
  • dyson-perrins-1.frodo.ox.ac.uk
  • earth-sciences.frodo.ox.ac.uk
  • egrove-park.frodo.ox.ac.uk
  • ertegun-house.frodo.ox.ac.uk
  • exeter-college.frodo.ox.ac.uk
  • experimental-psychology.frodo.ox.ac.uk
  • florey-building.frodo.ox.ac.uk
  • fmrib.frodo.ox.ac.uk
  • frewin-hall.frodo.ox.ac.uk
  • green-shed.frodo.ox.ac.uk
  • green-templeton-college.frodo.ox.ac.uk
  • hands-building.frodo.ox.ac.uk
  • harcourt-arboretum.frodo.ox.ac.uk
  • harkness-1.frodo.ox.ac.uk
  • herbert-close.frodo.ox.ac.uk
  • hertford-college.frodo.ox.ac.uk
  • history-of-science.frodo.ox.ac.uk
  • hollybush-row.frodo.ox.ac.uk
  • holywell-manor.frodo.ox.ac.uk
  • hume-rothery.frodo.ox.ac.uk
  • information-engineering.frodo.ox.ac.uk
  • isis-guest-house.frodo.ox.ac.uk
  • james-mellon-hall.frodo.ox.ac.uk
  • jenkin.frodo.ox.ac.uk
  • jesus-college.frodo.ox.ac.uk
  • jowett-walk.frodo.ox.ac.uk
  • keble-college.frodo.ox.ac.uk
  • kellogg-college.frodo.ox.ac.uk
  • kennedy.frodo.ox.ac.uk
  • king-charles-house.frodo.ox.ac.uk
  • lady-margaret-hall.frodo.ox.ac.uk
  • lampl-building.frodo.ox.ac.uk
  • language-teaching.frodo.ox.ac.uk
  • liddell-building-2.frodo.ox.ac.uk
  • liddell-building.frodo.ox.ac.uk
  • linacre-college.frodo.ox.ac.uk
  • lincoln-college.frodo.ox.ac.uk
  • magdalen-college.frodo.ox.ac.uk
  • maison-francaise.frodo.ox.ac.uk
  • malthouse.frodo.ox.ac.uk
  • manor-road-2.frodo.ox.ac.uk
  • mansfield-college.frodo.ox.ac.uk
  • mdx-62-banbury-road.frodo.ox.ac.uk
  • mdx-christchurch.frodo.ox.ac.uk
  • mdx-daubney.frodo.ox.ac.uk
  • mdx-engineering.frodo.ox.ac.uk
  • mdx-ewert.frodo.ox.ac.uk
  • mdx-merton.frodo.ox.ac.uk
  • mdx-plant-sciences.frodo.ox.ac.uk
  • mdx-social-studies.frodo.ox.ac.uk
  • mdx-st-cross.frodo.ox.ac.uk
  • medawar.frodo.ox.ac.uk
  • medical-oncology.frodo.ox.ac.uk
  • merifield.frodo.ox.ac.uk
  • merton-college.frodo.ox.ac.uk
  • merton-sports-ground.frodo.ox.ac.uk
  • miller-building.frodo.ox.ac.uk
  • mission-studies.frodo.ox.ac.uk
  • mstc.frodo.ox.ac.uk
  • music-faculty.frodo.ox.ac.uk
  • natural-history.frodo.ox.ac.uk
  • new-college.frodo.ox.ac.uk
  • new-richards.frodo.ox.ac.uk
  • nissan-institute.frodo.ox.ac.uk
  • nuffield-college.frodo.ox.ac.uk
  • ocdem.frodo.ox.ac.uk
  • ocgf.frodo.ox.ac.uk
  • old-bodleian-library.frodo.ox.ac.uk
  • old-boys-high-school.frodo.ox.ac.uk
  • old-comet-warehouse.frodo.ox.ac.uk
  • old-rectory.frodo.ox.ac.uk
  • orc-security.frodo.ox.ac.uk
  • orc-undercroft.frodo.ox.ac.uk
  • oxcis.frodo.ox.ac.uk
  • oxford-internet-institute.frodo.ox.ac.uk
  • oxford-university-press.frodo.ox.ac.uk
  • pembroke-college.frodo.ox.ac.uk
  • physical-chemistry.frodo.ox.ac.uk
  • pitt-rivers.frodo.ox.ac.uk
  • psychiatry.frodo.ox.ac.uk
  • queen-elizabeth-house.frodo.ox.ac.uk
  • radcliffe-camera.frodo.ox.ac.uk
  • radcliffe-house.frodo.ox.ac.uk
  • radcliffe-outpatients.frodo.ox.ac.uk
  • radcliffe-science-library.frodo.ox.ac.uk
  • regents-park-college.frodo.ox.ac.uk
  • rewley-abbey-court.frodo.ox.ac.uk
  • rewley-house.frodo.ox.ac.uk
  • rhodes-house.frodo.ox.ac.uk
  • robert-hooke-1.frodo.ox.ac.uk
  • robert-saunders-house.frodo.ox.ac.uk
  • roq-security.frodo.ox.ac.uk
  • rothermere-institute.frodo.ox.ac.uk
  • sackler-library.frodo.ox.ac.uk
  • said-business-school.frodo.ox.ac.uk
  • savile-house.frodo.ox.ac.uk
  • sers.frodo.ox.ac.uk
  • sheldonian.frodo.ox.ac.uk
  • somerville-college.frodo.ox.ac.uk
  • south-lodge.frodo.ox.ac.uk
  • southwell.frodo.ox.ac.uk
  • speedwell-house.frodo.ox.ac.uk
  • sports-centre.frodo.ox.ac.uk
  • stanford-college.frodo.ox.ac.uk
  • st-annes-college.frodo.ox.ac.uk
  • st-antonys-college.frodo.ox.ac.uk
  • statistics.frodo.ox.ac.uk
  • st-benets-hall.frodo.ox.ac.uk
  • st-catherines-college.frodo.ox.ac.uk
  • st-cross-college.frodo.ox.ac.uk
  • st-edmund-hall.frodo.ox.ac.uk
  • stevens-close.frodo.ox.ac.uk
  • st-hildas-college-2.frodo.ox.ac.uk
  • st-hildas-college.frodo.ox.ac.uk
  • st-hughs-college.frodo.ox.ac.uk
  • st-johns-college.frodo.ox.ac.uk
  • st-lukes-chapel.frodo.ox.ac.uk
  • st-peters-college.frodo.ox.ac.uk
  • st-stephens-house.frodo.ox.ac.uk
  • summertown-house.frodo.ox.ac.uk
  • tentorium.frodo.ox.ac.uk
  • the-queens-college.frodo.ox.ac.uk
  • thom.frodo.ox.ac.uk
  • trinity-college.frodo.ox.ac.uk
  • tubney-woods.frodo.ox.ac.uk
  • university-church.frodo.ox.ac.uk
  • university-club.frodo.ox.ac.uk
  • university-college-2.frodo.ox.ac.uk
  • university-college.frodo.ox.ac.uk
  • wadham-college.frodo.ox.ac.uk
  • warneford.frodo.ox.ac.uk
  • warnock-house.frodo.ox.ac.uk
  • waynflete.frodo.ox.ac.uk
  • weston-buildings.frodo.ox.ac.uk
  • williams-college.frodo.ox.ac.uk
  • winchester-house.frodo.ox.ac.uk
  • wolfson-building.frodo.ox.ac.uk
  • wolfson-college.frodo.ox.ac.uk
  • worcester-college.frodo.ox.ac.uk
  • wycliffe-hall.frodo.ox.ac.uk
  • wytham-field-station.frodo.ox.ac.uk
  • wytham-woods.frodo.ox.ac.uk

third batch

To be scheduled for morning 5th March 2020 modulo anything arising after second batch migrations.

  • 11-pusey-lane.frodo.ox.ac.uk
  • 13-banbury-road.frodo.ox.ac.uk
  • 13-norham-gardens.frodo.ox.ac.uk
  • 14-parks-road.frodo.ox.ac.uk
  • 15-norham-gardens.frodo.ox.ac.uk
  • 1-south-parks-road.frodo.ox.ac.uk
  • 2-south-parks-road.frodo.ox.ac.uk
  • 41-wellington-square.frodo.ox.ac.uk
  • 45-banbury-road.frodo.ox.ac.uk
  • 49-walton-street.frodo.ox.ac.uk
  • 4-worcester-street.frodo.ox.ac.uk
  • 5-worcester-street.frodo.ox.ac.uk
  • 66-banbury-road.frodo.ox.ac.uk
  • 6-worcester-street.frodo.ox.ac.uk
  • all-souls-college.frodo.ox.ac.uk
  • beaver-house.frodo.ox.ac.uk
  • begbroke-farmhouse.frodo.ox.ac.uk
  • begbroke-iat.frodo.ox.ac.uk
  • belsyre-court.frodo.ox.ac.uk
  • biochemistry.frodo.ox.ac.uk
  • bioescalator-1.frodo.ox.ac.uk
  • botnar.frodo.ox.ac.uk
  • boundary-brook-house.frodo.ox.ac.uk
  • buxton-court.frodo.ox.ac.uk
  • castle-mill-1.frodo.ox.ac.uk
  • ccmp.frodo.ox.ac.uk
  • clarendon-building.frodo.ox.ac.uk
  • clarendon-institute.frodo.ox.ac.uk
  • dartington-house.frodo.ox.ac.uk
  • dist-62-banbury-road.frodo.ox.ac.uk
  • dist-ashmolean.frodo.ox.ac.uk
  • dist-ewert-house.frodo.ox.ac.uk
  • dist-st-hughs.frodo.ox.ac.uk
  • dyson-perrins-2.frodo.ox.ac.uk
  • eagle-house.frodo.ox.ac.uk
  • eng-and-tech.frodo.ox.ac.uk
  • ewert-house.frodo.ox.ac.uk
  • exam-schools.frodo.ox.ac.uk
  • frewin-court.frodo.ox.ac.uk
  • gibson.frodo.ox.ac.uk
  • harris-manchester-college.frodo.ox.ac.uk
  • hayes-house.frodo.ox.ac.uk
  • hb-allen-centre.frodo.ox.ac.uk
  • holder.frodo.ox.ac.uk
  • inorganic-chemistry.frodo.ox.ac.uk
  • john-radcliffe-1.frodo.ox.ac.uk
  • john-radcliffe-2.frodo.ox.ac.uk
  • le-gros-clark.frodo.ox.ac.uk
  • life-sciences.frodo.ox.ac.uk
  • littlegate-house.frodo.ox.ac.uk
  • manor-road.frodo.ox.ac.uk
  • mdx-ashmolean.frodo.ox.ac.uk
  • mdx-orc.frodo.ox.ac.uk
  • mdx-st-hughs.frodo.ox.ac.uk
  • mdx-usdc.frodo.ox.ac.uk
  • ndm.frodo.ox.ac.uk
  • oerc-1.frodo.ox.ac.uk
  • oerc-2.frodo.ox.ac.uk
  • old-indian-institute.frodo.ox.ac.uk
  • old-observatory.frodo.ox.ac.uk
  • ompi.frodo.ox.ac.uk
  • orcrb-1.frodo.ox.ac.uk
  • oriel-college.frodo.ox.ac.uk
  • osney-1.frodo.ox.ac.uk
  • pharmacology.frodo.ox.ac.uk
  • plant-sciences-1.frodo.ox.ac.uk
  • plant-sciences-2.frodo.ox.ac.uk
  • radcliffe-infirmary.frodo.ox.ac.uk
  • radiology.frodo.ox.ac.uk
  • rex-richards.frodo.ox.ac.uk
  • robert-hooke-2.frodo.ox.ac.uk
  • rodney-porter.frodo.ox.ac.uk
  • sherrington.frodo.ox.ac.uk
  • st-cross-building.frodo.ox.ac.uk
  • st-cross-road-annexe.frodo.ox.ac.uk
  • taylorian-library.frodo.ox.ac.uk
  • tinsley.frodo.ox.ac.uk
  • university-offices-1.frodo.ox.ac.uk
  • university-offices-2.frodo.ox.ac.uk
  • weston-library.frodo.ox.ac.uk
  • wolfson-jr.frodo.ox.ac.uk
Posted in Uncategorized | Leave a comment

OWL Visitor refresh

This post aims to give a bit more background and detail around the announcement to IT Support Staff regarding the forthcoming refresh to the OWL Visitor service.

What are we doing?

We’re replacing the captive portal component of the service.  This is the part that shows the visitor login page as well handling things like the firewall rules and NAT.

The images below show the current (l) and new (r) portal login pages.

We are not changing anything else about the service so the account management system, administration of account administrators and visitor RADIUS servers all remain the same.

 

When are we doing it?

The main roll out is scheduled to take place on 3rd March 2020.

IT Services has been running the new OWL since 14th January, so pop along to 13 Banbury Road if you want to take an early look.

We are inviting ITSS to help us test the new system from early February so please let us know if you’d like to take part.  (The networks team has no access to test clients and so has only been able to test with what its staff members happen to use).

 

Why are we doing it?

  1. The current OWL portal has been running for 15 years.  That’s over 100 in dog years and it has a number of limitations commensurate with its age.
  2. The University rejected JISC’s eduroam Visitor Access service as an OWL replacement on Information Security grounds.
  3. The OWL portal doesn’t support TLS versions above 1.0 (see point 1).  This will become an issue in March when the major browser vendors start dropping support for TLS 1.0.

 

How are we doing it?

A new VRF has been deployed across Odin (campus backbone) with its gateway being a new pair of servers running pfSense.  On changeover day, your OWL FroDo port is simply associated with the new VRF.  This allows us to easily roll out on a FroDo by FroDo basis and revert if necessary.

 

Posted in Wireless | Leave a comment

Hydra – a new DNS/IPAM platform

We are pleased to announce the forthcoming launch of Hydra – our new DNS and IP Address Management platform.

This will replace the current DNS web management tool.

 

Timetable

The last DNS update for the current system will be at 18:00 on Friday, 29 November 2019.

Hydra will go live during the morning of Tuesday, 03 December 2019.

There will be an Early Life Support period until the end of January 2020 during which the Hostmaster team will make edits on your behalf.

 

Things to do now as ITSS01

Check the lists of domains and subnets to make sure your Unit owns all the resources you think it should.  There’s also a handy summary list.  Please contact <networks@it.ox.ac.uk> to claim any you believe you should own.

Unless you’re happy to be the only person able to edit your Unit’s DNS records, you’ll need to gain access to Groupstore so that you can edit the your Unit’s members group.

Read the help pages and also play in the sandpit.  The sandpit uses a non-production database so you can try things out without worrying about affecting live data.

Book onto one of two ITSS sessions (15 and 24 October) where you can ask any questions you may have.

 

Things to do now as ITSS

Ask your ITSS01 to add you to your Unit’s Groupstore IPAM Group if you want to be able to edit DNS records.

Read the help pages and also play in the sandpit.  The sandpit uses a non-production database so you can try things out without worrying about affecting live data.

Book onto one of two ITSS sessions (15 and 24 October) where you can ask any questions you may have.

 

Posted in DNS | Leave a comment

DC Distributor FroDo – Upgrades November 2019

Introduction      

We would like to announce the upgrade of the version of Comware running on our DC Distributor FroDos. This follows on from the recent upgrade of all customer FroDos during September and the start of this month.

This blog entry aims to answer the majority of questions that this work will raise. Please, however, feel free to contact the Networks team with any further questions at networks@it.ox.ac.uk

Why?

As part of ongoing maintenance it is essential that we keep our FroDo software up to date. The new version of software being deployed addresses a number of vulnerabilities and bugs. This takes us from R2612H01 to R2702 for these devices.

Impact

All DC Distributor FroDos are HPE 5940’s running in an IRF pair. As a result, we can use the HPE In Service Software Update (ISSU) feature which aims to minimise impact to traffic passing through the FroDo pair as it is upgraded.

The DC distributors do run a variety of customer annexe connections into the DC sites Frodo and this information is listed within Huggin. Just enter the name or frodo number. Any customers who do not have an aggregated annexe connection across both devices will be affected as the single device they connect to reloads. The reload time for a HPE 5940 is ~10 minutes.

Finally, we have recently migrated several services from the old service spine onto pairs of DC distributors. With that in mind I have provided a list, in upgrade order, of the central services that are hosted on each device.

It is important to remember that we have resilience in the form of IRF as well as the service being provided from two distinct dcdist FroDos. However, as with all upgrades, there is an element of risk and therefore the services themselves should be considered at risk during the upgrade window.

Upgrade Schedule

Tuesday 5th November

dcdist-usdc-1 (frodo-120809) (07:00)

ESTSN(ASA)
COWLS WLC USDC

Thursday 7th November

 dcdist-edc (frodo-030403) (07:00)

OXIDE-ESTATES-FACILITIES#2
OXIDE-ESTATES-SECURE#2
OXIDE-CHORUS#2

dcdist-roq (frodo-050911) (07:30) This device rescheduled for 22/11 07:00

CENTRAL-VPN#1
EDUROAM#1
COWLS WLC ROQ
SMALL OFFICES (MNS)
SMALL ESTATES(MNS)
SMALL LIBRARIES

Tuesday 12th November

dcdist-beach(frodo-120601) (07:00)

OXIDE CHORUS#1
OXIDE ESTATES-FACILITIES#1
OXIDE ESTATES-SECURE#2

dcdist-orc(frodo-100912) (07:30)

OWL#1
COWLS WLC ORC

Thursday 14th November

 dcdist-ind(frodo-030812) (07:00)

EDUROAM#2
CENTRAL-VPN#2
COWLS WLC IND

dcdist-mus(frodo-120610) (07:30)

OWL#2
COWLS WLC MUS

Tues 19th November

dcdist-beg(frodo-050909) (07:00)
dcdist-osney(frodo-030811) (07:30)

 

Thursday 22nd November

dcdist-roq (frodo-050911) (07:00)

CENTRAL-VPN#1
EDUROAM#1
COWLS WLC ROQ
SMALL OFFICES (MNS)
SMALL ESTATES(MNS)
SMALL LIBRARIES

Posted in General Maintenance, Odin | Leave a comment

Frodo Upgrades 2019

FroDo Comware Upgrade

We would like to announce a staged upgrade of the version of Comware running on our HPE 5510 and 5940 FroDos. This blog entry aims to answer the majority of questions that this work will raise. Please, however, feel free to contact the Networks team with any further questions at networks@it.ox.ac.uk

NOTE: This does not include upgrading the dcdist FroDos – these will be upgraded as a separate task in due course.

Why?

As part of ongoing maintenance it is essential that we keep our FroDo software up to date. The new versions of software being deployed address a number of vulnerabilities and bugs. For those interested this upgrade takes us from R1309P06 to R1311P02 for HPE 5510 devices and R2612H01 to R2702 for HPE 5940s. In total this change involves over 330 devices.

Addressed Vulnerabilities

201811140403

  • Symptom: CVE-2018-15473

Condition: OpenSSH is prone to a user-enumeration vulnerability. An attacker may leverage this issue to harvest valid user accounts, which may aid in brute-force attacks. OpenSSH through 7.7 are vulnerable; other versions may also be affected.

 

Information about the detail of vulnerabilities can be found at https://cve.mitre.org/cve/search_cve_list.html

 

Impact

The expected impact is ~5-10 minutes for Option 1 customers during which time the FroDo will reload and external services will not be available. For Option 2 customers the impact is expected to be minimal thanks to the In Service Software Upgrade (ISSU) capability.

We will be carrying out the upgrades between 06:30 and 08:00 to minimise impact.

Timescale

We plan to upgrade approximately 80 FroDo’s on each of the following days:

Group A: Thursday 19th September
Group B: Tuesday 24th September
Group C: Thursday 26th September
Group D: Tuesday 1st October

Schedule

 

We have attempted, where possible, to group devices around main sites and annexes so that those sites will only see one period of disruption from this upgrade schedule. Detailed schedules listing devices and dates can be found at https://docs.ntg.ox.ac.uk/pub/reference/odin-frodo-software-upgrade-september-october-2019

 

Once again, if you have any further queries then please contact us at networks@it.ox.ac.uk

 

Posted in Uncategorized | Leave a comment

What happens to mail you mark as spam?

This blog post hasn’t spun out of any particular instance, but I sometimes get the feeling that clicking the “Mark as SPAM” button in a mail client isn’t completely understood by everyone, which isn’t surprising as everything that follows is what could happen rather than what actually does happen. These scenarios are up to a combination of sending domain and receiving domain policies to bring into force. The take-home message should be to not mark legitimate emails as SPAM. If it’s a mailing list, then unsubscribe; if it’s a hot email thread involving your co-workers, create an inbox rule to mute the thread; if it’s incessant “shared” emails from a relative ask them to stop!

Without further ado, this is what could happen if you mark an email as SPAM:

The email gets deleted in your mailbox

The starter for 10; there’s little point in you marking an email as SPAM if it didn’t get removed from your INBOX. I’m sure many people have clicking the SPAM button and the Delete button as equivalent actions. One particularly nasty UI decision of a web based mail client once required a single click to mark an email as spam, but two to delete the email. Of course people took the path of less resistance and clicked the SPAM button rather than the delete button, which wasn’t great because of the scenarios that follow.

The sending domain/server gets blocked or similar emails to other recipients get emails sent to junk

One person’s SPAM is another person’s genuine enhancement pills newsletter. OK, an email advertising stamina enhancing drugs is perhaps a silly example, but if you’re getting regular emails from a mailing list, you may be doing a disservice to other subscribers who want to receive these emails when you mark them as SPAM. Some mail providers handle this better than others, but there is a risk that if you mark an email as spam when it isn’t, other’s will not receive the email into their INBOX.

Postmaster may get a copy of your email

This is probably the least known aspect of SPAM management. When you mark an email as such, the sending domain potentially has the ability to request information about the spam message, including the full message’s contents. You will have to read the T&Cs of your mail provider, but I wouldn’t be surprised if yours has a clause saying that mail you marked as SPAM has different privacy levels as your legitimate mail.

How does this forwarding back to the sending domain happen? This varies based on receiving domain. Using Microsoft as an example of how you can do it, they have a Junk Mail Reporting Program (JMRP) where you can enroll your sending servers. Any email marked as spam that was sent from these servers is sent a copy to a configurable email address.

Now that the sending server has a copy of the “SPAM”, how is it analyzed? I’ll leave that to your imagination, but not every company has the ability to use just artificial intelligence. Some human interaction may be involved.

Conclusion

Don’t mark genuine email as SPAM! It’s in everyone’s interest.

Posted in Uncategorized | Leave a comment

The Clam Closes

We use a lot of open source software in our team and we try to contribute a little back to the community when we can.  The central mail relay, Oxmail, had been using ClamAV since sometime between 2003 and 2005 and when we discovered that we could host a public mirror of the signature databases we set one up in 2007.

This was an apache vhost on our team webserver running on a trusty IBM eServer xSeries 336.  That server was only recently decommissioned after having given 12 years of faultless production service.  In 2013, the mirror was moved to a dedicated webserver running in a VM.

The ClamAV project was acquired by Sourcefire in 2007, which itself was acquired by Cisco in 2013.  Over the summer, Cisco changed the DNS records that clients should use to find a mirror to point to Cloudflare’s content delivery network.  Our mirror still received thousands of hits per day from clients that had presumably hard-coded our mirror’s IP address in their config.  We recently learnt that Cisco had silently stopped updating the signature databases on volunteer mirrors and so our mirror was serving stale data.  We considered it better to stop serving altogether rather than to give clients out-of-date signatures and so switched off our mirror today.

In our busiest month over the past 11 years of service, our mirror served up 17 TB of data at a peak transfer rate of 8 Gb/s.

Posted in Services | Tagged , | Leave a comment

September 2018 – Odin 5940 FroDo Comware Upgrade – Additional full reboots required

From both my last post and my colleague Rob Perkins’ previous post, you’ll see that we’ve had some fun and games recently with updating the software on the FroDos provisioned on the HPE 5940 platform.

Whilst these FroDos represent a relatively small proportion of the Odin FroDo estate (<10%), this has been enough to create a reasonable amount of work for us (and I imagine for you as ITSS also). Sadly this has also resulted in unplanned and unavoidable disruptions to Odin service for affected customers. For this we can only sincerely apologise and rest assured, we are feeding all of this back to HPE in an effort to improve the situation moving forward.

It should perhaps be noted that the vast majority of customers on the HPE 5510 platform (which also happens to be currently undergoing a software update – see my colleague Mike’s post) would not have been subject to the unplanned disruptions mentioned in this post.

 

So what went wrong?

Essentially nothing from a hard-line technical perspective. The update involves both a main code update and a ‘hot’ patch (the latter of these we were provided with by HPE support to fix numerous issues which are documented in our previous posts). There’s nothing particularly extraordinary about any of that.

However what is unusual perhaps, is that the (so called) hot patch actually addresses some resource issues we’ve been seeing with this platform which involves re-juggling TCAM memory allocation on the switch. This is to allocate more resources in favour of some features which were struggling before in our implementation (control plane stuff like PIM multicast routing and OSPFv3 for instance) away from others which we aren’t using.

What we didn’t know until during the update process and as part of the support cases we subsequently opened with HPE, was that only a full reboot would complete the upgrade properly. Sadly it also seems that HPE hadn’t documented this clearly in their release notes which we are working with them to resolve.

Because the aforementioned additional reboot in general hasn’t happened during the upgrades so far, the L2 annexe VSI connectivity problem some units have observed and other issues we’ve seen so far are the result of a lack of resources. This issue can only be resolved permanently via the full reboot.

 

What do you mean ‘full’ reboot?

So a full reboot in this context is a reload of all switches involved in an Odin FroDo provision simultaneously.

This means in practice that regardless of whether your unit opted for Odin provisioning options 0-1 (you have only one switch operating as your FroDo) or if you opted for option 2 (you have two switches logically operating together in an IRF to act as one for resiliency purposes), your 5940 FroDo (or FroDo pair) will be down entirely during the reboot cycle. For option 2 customers this is a rare event as most upgrades can be carried out using the In-Service Software Update (ISSU) capability (as was our original intention with this one).

If you’re unsure of what your unit opted for, then you can check via the Huginn portal here.

If you’re still unclear about what the Odin provisioning options are, or what they mean, you should consult the Odin SLD and associated information here.

 

So what’s the plan moving forward?

A small number of 5940 FroDos have had their upgrade and full reboots already.

The remaining ones will need to have their full reboot and this is scheduled as follows:

Thursday 11th October
frodo-030809    dcdist-br           - 7.00am
frodo-100907    welcome-trust       - 7.00am
frodo-120601    beach-2             - 7.30am
frodo-100909    orcrb-2             - 7.30am
 
Tuesday 16th October
frodo-120809    dcdist-usdc         - 7.00am
frodo-120810    molecular-medicine  - 7.00am
frodo-030811    dcdist-osney        - 7.30am

Impact

The expected outage whilst each reboot completes is approximately 10 minutes.

 

Is this really necessary?

Unfortunately yes. We’ve weighed up the potential consequences of doing nothing vs undertaking the additional reboots and we just aren’t comfortable with the former. This is because doing nothing has the potential to introduce difficult to diagnose issues resulting from potential TCAM exhaustion later on.

Posted in Uncategorized | Leave a comment

September 2018 – Odin 5940 Frodo Upgrade – Take 2

Odin 5940 FroDo Comware Upgrade (reattempt)

We would like to announce a staged upgrade of the version of Comware running on our HPE 5940 FroDos for those that were not completed last time around. This blog entry aims to answer the majority of questions that this work will raise. Please, feel free to contact the Networks team with any further questions at networks@it.ox.ac.uk

What Happened Last time & Remediation Steps Moving Forward

Essentially we encountered an unexpected issue the last time around with unit L2 annexe connectivity not being re-established following the application of a hot patch which is part of the upgrade. This is strange as the MAC learning continues to work which initially gave us the impression last time around that all was well. This issue is logged as a support case with our vendor HPE and unfortunately to date, they’ve been unable to replicate the issue we had. We’ll therefore be seeking their availability on a remote session for at least one option 1 and option 2 upgrade to ensure that if the issue recurs we can get their eyeballs on to it.

In the meantime, we have a workaround which is to ‘turn it off and turn it on again’. Seriously, should the issue recur the workaround is to shut down the L2VPN Virtual Switching Instance (VSI) serving the annexe connection on the affected FroDo and then re-enable it which we’ll do in instances should it proves necessary to re-establish connectivity post-upgrade.

Why?

I shan’t be repeating what Rob Perkins said in his original post. If you’d like to know why this upgrade is needed, please read his original post here.

Impact

The expected impact is ~5-10 minutes for Option 1 customers during which time the FroDo will reload and external services will not be available. For Option 2 customers the impact is expected to be minimal thanks to the In Service Software Upgrade (ISSU) capability.

Because we wish to get this completed before the start of Michaelmas term, we will be carrying out the upgrades as per the  accelerated schedule below. Please accept our apologies for any inconvenience caused by these upgrades.

Timescale

Thursday 20th September
frodo-120601 beach-2 - 7.30am
frodo-120809 dcdist-usdc (option 2) - 8.00am

Tuesday 25th September
frodo-120810 molecular-medicine - 7.00am
frodo-050909 dcdist-beg (formerly begbroke-iat-1 - option 2) - 7.30am

Wednesday 26th September
frodo-120812 john-radcliffe-3 - 7.00am
frodo-100908 richard-doll (option 2) - 7.30am
frodo-120811 big-data-institute (option 2) - 8.00am

 

Posted in Uncategorized | Leave a comment

September 2018 Odin FroDo Upgrade

FroDo Comware Upgrade

We would like to announce a staged upgrade of the version of Comware running on our HPE 5510 FroDos. This blog entry aims to answer the majority of questions that this work will raise. Please, however,  feel free to contact the Networks team with any further questions at networks@it.ox.ac.uk

Why?

As part of ongoing maintenance it is essential that we keep our FroDo software up to date. The new version of software being deployed addresses a number of vulnerabilities and bugs. For those interested this upgrade takes us from R1309 to R1309 P06 and involves over 300 devices.

Relevant Bug Fixes

201806290399
• Symptom: The value of the snmpEngineboot node is incorrect.
• Condition: This symptom occurs if the whole IRF fabric is rebooted to cause a master/subordinate switchover.

Addressed Vulnerabilities

This release addresses the following CVE

CVE-2016-9586
CVE-2017-15896
CVE-2017-3737
CVE-2017-3738
CVE-2017-3736
CVE-2017-12190
CVE-2017-12192
CVE-2017-15274
CVE-2017-15299
CVE-2017-1000253
CVE-2017-3735
CVE-2017-6458
CVE-2016-9042
CVE-2014-9297
CVE-2015-9298

Information about the detail of these vulnerabilities can be found at https://cve.mitre.org/cve/search_cve_list.html

Impact

The expected impact is ~5-10 minutes for Option 1 customers during which time the FroDo will reload and external services will not be available. For Option 2 customers the impact is expected to be minimal thanks to the In Service Software Upgrade (ISSU) capability introduced in the firmware update applied in August 2017.

We will be carrying out the upgrades between 06:00 and 07:30 to minimise impact.

Timescale

We plan to upgrade approximately 80 FroDo’s on the each of the following days:

Group A: Tuesday 18th September
Group B: Thursday 20th September
Group C: Tuesday 25th September
Group D: Thursday 27th September

Schedule

We have attempted,where possible, to group devices around main sites and annexes so that those sites will only see one period of disruption. Detailed schedules listing devices and dates can be found at https://docs.ntg.ox.ac.uk/pub/reference/FroDoUpgrade-Sep2018

Once again, if you have any further queries then please contact us at networks@it.ox.ac.uk

Posted in General Maintenance, Odin | Leave a comment