FroDo Comware Upgrade

We would like to announce a staged upgrade of the version of Comware running on our HPE 5510 and 5940 FroDos. This blog entry aims to answer the majority of questions that this work will raise. Please, however, feel free to contact the Networks team with any further questions at networks@it.ox.ac.uk

NOTE: This does not include upgrading the dcdist FroDos – these will be upgraded as a separate task in due course.

Why?

As part of ongoing maintenance it is essential that we keep our FroDo software up to date. The new versions of software being deployed address a number of vulnerabilities and bugs. For those interested this upgrade takes us from R1309P06 to R1311P02 for HPE 5510 devices and R2612H01 to R2702 for HPE 5940s. In total this change involves over 330 devices.

Addressed Vulnerabilities

201811140403

• Symptom: CVE-2018-15473

Condition: OpenSSH is prone to a user-enumeration vulnerability. An attacker may leverage this issue to harvest valid user accounts, which may aid in brute-force attacks. OpenSSH through 7.7 are vulnerable; other versions may also be affected.

Information about the detail of vulnerabilities can be found at https://cve.mitre.org/cve/search_cve_list.html

Impact

The expected impact is ~5-10 minutes for Option 1 customers during which time the FroDo will reload and external services will not be available. For Option 2 customers the impact is expected to be minimal thanks to the In Service Software Upgrade (ISSU) capability.

We will be carrying out the upgrades between 06:30 and 08:00 to minimise impact.

Timescale

We plan to upgrade approximately 80 FroDo’s on each of the following days:

Group A: Thursday 19^th September
Group B: Tuesday 24^th September
Group C: Thursday 26^th September
Group D: Tuesday 1^st October

Schedule

We have attempted, where possible, to group devices around main sites and annexes so that those sites will only see one period of disruption from this upgrade schedule. Detailed schedules listing devices and dates can be found at https://docs.ntg.ox.ac.uk/pub/reference/odin-frodo-software-upgrade-september-october-2019

Once again, if you have any further queries then please contact us at networks@it.ox.ac.uk

This blog post hasn’t spun out of any particular instance, but I sometimes get the feeling that clicking the “Mark as SPAM” button in a mail client isn’t completely understood by everyone, which isn’t surprising as everything that follows is what could happen rather than what actually does happen. These scenarios are up to a combination of sending domain and receiving domain policies to bring into force. The take-home message should be to not mark legitimate emails as SPAM. If it’s a mailing list, then unsubscribe; if it’s a hot email thread involving your co-workers, create an inbox rule to mute the thread; if it’s incessant “shared” emails from a relative ask them to stop!

Without further ado, this is what could happen if you mark an email as SPAM:

The email gets deleted in your mailbox

The starter for 10; there’s little point in you marking an email as SPAM if it didn’t get removed from your INBOX. I’m sure many people have clicking the SPAM button and the Delete button as equivalent actions. One particularly nasty UI decision of a web based mail client once required a single click to mark an email as spam, but two to delete the email. Of course people took the path of less resistance and clicked the SPAM button rather than the delete button, which wasn’t great because of the scenarios that follow.

The sending domain/server gets blocked or similar emails to other recipients get emails sent to junk

One person’s SPAM is another person’s genuine enhancement pills newsletter. OK, an email advertising stamina enhancing drugs is perhaps a silly example, but if you’re getting regular emails from a mailing list, you may be doing a disservice to other subscribers who want to receive these emails when you mark them as SPAM. Some mail providers handle this better than others, but there is a risk that if you mark an email as spam when it isn’t, other’s will not receive the email into their INBOX.

Postmaster may get a copy of your email

This is probably the least known aspect of SPAM management. When you mark an email as such, the sending domain potentially has the ability to request information about the spam message, including the full message’s contents. You will have to read the T&Cs of your mail provider, but I wouldn’t be surprised if yours has a clause saying that mail you marked as SPAM has different privacy levels as your legitimate mail.

How does this forwarding back to the sending domain happen? This varies based on receiving domain. Using Microsoft as an example of how you can do it, they have a Junk Mail Reporting Program (JMRP) where you can enroll your sending servers. Any email marked as spam that was sent from these servers is sent a copy to a configurable email address.

Now that the sending server has a copy of the “SPAM”, how is it analyzed? I’ll leave that to your imagination, but not every company has the ability to use just artificial intelligence. Some human interaction may be involved.

Conclusion

Don’t mark genuine email as SPAM! It’s in everyone’s interest.

From both my last post and my colleague Rob Perkins’ previous post, you’ll see that we’ve had some fun and games recently with updating the software on the FroDos provisioned on the HPE 5940 platform.

Whilst these FroDos represent a relatively small proportion of the Odin FroDo estate (<10%), this has been enough to create a reasonable amount of work for us (and I imagine for you as ITSS also). Sadly this has also resulted in unplanned and unavoidable disruptions to Odin service for affected customers. For this we can only sincerely apologise and rest assured, we are feeding all of this back to HPE in an effort to improve the situation moving forward.

It should perhaps be noted that the vast majority of customers on the HPE 5510 platform (which also happens to be currently undergoing a software update – see my colleague Mike’s post) would not have been subject to the unplanned disruptions mentioned in this post.

So what went wrong?

Essentially nothing from a hard-line technical perspective. The update involves both a main code update and a ‘hot’ patch (the latter of these we were provided with by HPE support to fix numerous issues which are documented in our previous posts). There’s nothing particularly extraordinary about any of that.

However what is unusual perhaps, is that the (so called) hot patch actually addresses some resource issues we’ve been seeing with this platform which involves re-juggling TCAM memory allocation on the switch. This is to allocate more resources in favour of some features which were struggling before in our implementation (control plane stuff like PIM multicast routing and OSPFv3 for instance) away from others which we aren’t using.

What we didn’t know until during the update process and as part of the support cases we subsequently opened with HPE, was that only a full reboot would complete the upgrade properly. Sadly it also seems that HPE hadn’t documented this clearly in their release notes which we are working with them to resolve.

Because the aforementioned additional reboot in general hasn’t happened during the upgrades so far, the L2 annexe VSI connectivity problem some units have observed and other issues we’ve seen so far are the result of a lack of resources. This issue can only be resolved permanently via the full reboot.

What do you mean ‘full’ reboot?

So a full reboot in this context is a reload of all switches involved in an Odin FroDo provision simultaneously.

This means in practice that regardless of whether your unit opted for Odin provisioning options 0-1 (you have only one switch operating as your FroDo) or if you opted for option 2 (you have two switches logically operating together in an IRF to act as one for resiliency purposes), your 5940 FroDo (or FroDo pair) will be down entirely during the reboot cycle. For option 2 customers this is a rare event as most upgrades can be carried out using the In-Service Software Update (ISSU) capability (as was our original intention with this one).

If you’re unsure of what your unit opted for, then you can check via the Huginn portal here.

If you’re still unclear about what the Odin provisioning options are, or what they mean, you should consult the Odin SLD and associated information here.

So what’s the plan moving forward?

A small number of 5940 FroDos have had their upgrade and full reboots already.

The remaining ones will need to have their full reboot and this is scheduled as follows:

Thursday 11th October
frodo-030809    dcdist-br           - 7.00am
frodo-100907    welcome-trust       - 7.00am
frodo-120601    beach-2             - 7.30am
frodo-100909    orcrb-2             - 7.30am
 
Tuesday 16th October
frodo-120809    dcdist-usdc         - 7.00am
frodo-120810    molecular-medicine  - 7.00am
frodo-030811    dcdist-osney        - 7.30am

Impact

The expected outage whilst each reboot completes is approximately 10 minutes.

Is this really necessary?

Unfortunately yes. We’ve weighed up the potential consequences of doing nothing vs undertaking the additional reboots and we just aren’t comfortable with the former. This is because doing nothing has the potential to introduce difficult to diagnose issues resulting from potential TCAM exhaustion later on.

Odin 5940 FroDo Comware Upgrade (reattempt)

We would like to announce a staged upgrade of the version of Comware running on our HPE 5940 FroDos for those that were not completed last time around. This blog entry aims to answer the majority of questions that this work will raise. Please, feel free to contact the Networks team with any further questions at networks@it.ox.ac.uk

What Happened Last time & Remediation Steps Moving Forward

Essentially we encountered an unexpected issue the last time around with unit L2 annexe connectivity not being re-established following the application of a hot patch which is part of the upgrade. This is strange as the MAC learning continues to work which initially gave us the impression last time around that all was well. This issue is logged as a support case with our vendor HPE and unfortunately to date, they’ve been unable to replicate the issue we had. We’ll therefore be seeking their availability on a remote session for at least one option 1 and option 2 upgrade to ensure that if the issue recurs we can get their eyeballs on to it.

In the meantime, we have a workaround which is to ‘turn it off and turn it on again’. Seriously, should the issue recur the workaround is to shut down the L2VPN Virtual Switching Instance (VSI) serving the annexe connection on the affected FroDo and then re-enable it which we’ll do in instances should it proves necessary to re-establish connectivity post-upgrade.

Why?

I shan’t be repeating what Rob Perkins said in his original post. If you’d like to know why this upgrade is needed, please read his original post here.

Impact

The expected impact is ~5-10 minutes for Option 1 customers during which time the FroDo will reload and external services will not be available. For Option 2 customers the impact is expected to be minimal thanks to the In Service Software Upgrade (ISSU) capability.

Because we wish to get this completed before the start of Michaelmas term, we will be carrying out the upgrades as per the  accelerated schedule below. Please accept our apologies for any inconvenience caused by these upgrades.

Timescale

Thursday 20th September
frodo-120601 beach-2 - 7.30am
frodo-120809 dcdist-usdc (option 2) - 8.00am

Tuesday 25th September
frodo-120810 molecular-medicine - 7.00am
frodo-050909 dcdist-beg (formerly begbroke-iat-1 - option 2) - 7.30am

Wednesday 26th September
frodo-120812 john-radcliffe-3 - 7.00am
frodo-100908 richard-doll (option 2) - 7.30am
frodo-120811 big-data-institute (option 2) - 8.00am