Outlook 2016 slow when connecting to Nexus Exchange

We’ve had reports from some Nexus users that Outlook 2016 can appear to hang while trying to make a connection to our Exchange servers. The reported delay is between thirty and forty seconds, after which the connection is established and normal service resumes.

People who have stayed on Outlook 2013 don’t generally encounter this issue, but if they do they can easily resolve it by tweaking settings.

Here is what’s going on: the Exchange 2010 Autodiscover service tells the client to try a regular RPC/TCP connection before resorting to a RPC/HTTP connection. 

In Outlook 2013 there is an option in the program’s settings: ‘On fast networks, connect using HTTP first, then connect using TCP/IP’. This setting resolves the issue (and these days ‘fast network’ means any connection that’s faster than dial-up).

But if you’ve updated to Outlook 2016 that option has disappeared. Essentially you’re looking at a mismatch of versions – Microsoft are assuming that Outlook 2016 will be connecting to Exchange 2016. Once we’ve migrated to Nexus365 we will be but, currently, we’re still on Exchange 2010 on-premises. In other words, even with our best-practice server configuration, newer versions of client software are creating ‘gotchas’ for us…

What can the Nexus Team do about this?

There is an option of mandating all Nexus client connections to use HTTP first. This is a server-side setting we can apply. However this has an adverse effect for everyone who doesn’t use Outlook. For us, that’s a lot of people. We have had to rule out that solution.

The longer-term solution is to migrate our users to Nexus365, since that will effectively bring the servers you are connecting to bang up to date. Pilot migrations begin next month.

What can I do about this?

The recommendation for University IT Support Staff is to use Group Policy to resolve this wherever possible – the policy settings that are equivalent to Outlook 2013’s tickboxes still exist. Microsoft may have removed the interface to see them in Outlook 2016 but the configuration can still be made, albeit via a circuitous route.

The setting you want to change is:

User Configuration\Administrative Templates\Microsoft Outlook 2016\Account Settings\Exchange

Enabling ‘flag 4’ is equivalent to ticking the checkbox in Outlook 2013 for using HTTP first.

Self-managing Outlook 2016 users can edit their registry to achieve the same effect. You need to create (or modify) this key:

HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\16.0\Outlook\RPC

Add a DWORD value of ‘ProxyServerFlags’ with a decimal value of 47.

Note that because this is a current-user setting you will also need to apply it for other users of the same PC.

If the problems persist, for multiple users, please verify that your DNS settings are correct for autodiscover to successfully resolve your unit’s subdomain. Even after all these years since Nexus went into service we still find occasional pockets of users putting up with slower-than-needed lookups and configuration challenges which are avoidable with autodiscover set correctly.


							
Posted in Uncategorized | Leave a comment

OnTheHub renewals…

When the University first signed up for the OnTheHub service plenty of University members went there to obtain free or discounted software. When asked to register, they used their official email address – after all, why wouldn’t you?

Some time later, as Nexus’ migration to Office 365 project kicked off, it became apparent that this couldn’t continue: we needed to reserve all of the University’s email domains/suffixes for use in our official Office 365 tenancy. Users who were using their University email addresses for any other interaction with Microsoft were creating a ‘viral tenancy’ with no administrative control, while simultaneously preventing us from adding that domain name to the official tenancy. And without the ability to add that domain name to ‘Nexus365’ that college or department would be stuck for all eternity on a tiny in-house Nexus mailbox.

It was around a year ago we identified this problem and asked Kivutu, who administer OnTheHub, to stop accepting registrations from University email addresses. Now users of OnTheHub are asked to register with an email address in the format first.last@oxforduni.onmicrosoft.com. Our help text was also updated to reflect that change.

A year on from that we now have a new problem. OnTheHub users are asked to periodically renew their licence, to prove that they are still members of the University and thus entitled to continue using that software.  It’s been a year, so University OnTheHub users are starting to relicence their software, as requested.

To verify entitlement, users are asked to log in…
The problem is that after all this time many users have forgotten about that oxforduni.onmicrosoft.com address – many of them have been trying to log in with their official University email address. Because we’ve been adding all of those email domains to the official Nexus365 tenancy, and we’ve federated it via Shibboleth to use single-sign-on addresses, Microsoft’s logon page sees an address that belongs to us and ‘helpfully’ sends the request in our direction.

Your end user sees a standard SSO logon, knows what to do with that, so logs in. However what they get is not OnTheHub. What they’re seeing is, in effect, a sneak preview of Nexus365. But it’s not usable in any meaningful sense as what they’re seeing is a not-yet-migrated and not-yet-licensed example of Nexus365. All of the new exciting toys are visible but none of them are usable until we are able to start migrating folk across.

It would therefore be a great help to us, and a useful reduction of the support burden on the helpdesk, if you could advise your OnTheHub users to look up their oxforduni.onmicrosoft.com alias and use that to re-licence their Microsoft software downloads. It will have been sent to their official University email address so should be searchable within their mailbox. Finding and using that address will  ensure that a greater number of people can continue using their software without delay or (additional) confusion by re-licensing at their first attempt.

 

 

 

Posted in Uncategorized | Leave a comment

Nexus365

The Nexus team are trying things out. My mailbox is now just a tad larger than it was yesterday.

Isn’t this number a lovely sight? 🙂

Posted in Uncategorized | Leave a comment

BlackBerry decline continues

The Nexus Blackberry Enterprise server is licensed for 378 BlackBerry devices. And, back in 2010, we needed every last one of those licences. Today things are very different. After a lengthy process contacting users and removing those who had given up their BlackBerries, there are now only 27 people still registered on the server. Of those, only 25 have made contact during the last month. This means that active usage has more than halved just since July 2016, when we counted sixty active users.

Nexus’ BlackBerry server software does not support the current range of BlackBerry handsets. In order to support these newer devices our server software would have to be upgraded. More importantly the version change requires users’ devices to be re-licensed (at cost). The expense and effort required to do this does not make good financial sense for a system hosting so few users. The service requires two BlackBerry Enterprise Servers, for redundancy, and a back-end SQL database. All of these need monitoring, updating, backup, and general fettling. For 25 people this routine upkeep doesn’t represent a good return on the effort required. The department would struggle to justify provision of any new service from which fewer than thirty University members would benefit and which only supports obsolete devices.

The intention is that Nexus’ BES service will be retired ahead of the migration to Office 365. All current BlackBerry server users (i.e. anyone who bought a licence and has gone through the server activation process) should plan to replace those devices as soon as possible. New Blackberry handsets can still be used to connect to Nexus but should be configured to connect only via the ActiveSync protocol. If you are a BlackBerry user who use BIS, or ActiveSync, to connect to Nexus you will be unaffected regardless of whether we maintain a Nexus BlackBerry Enterprise Service.

 

  • 81% are using a device that’s over five years old.
  • The other 19% have 9720 devices (which were first introduced in the summer of 2013).
  • The oldest devices in use – an 8310 and an 8800 model – date from 2007.
Posted in Uncategorized | Leave a comment

Nexus’ BlackBerry Enterprise Server

Early last week an email was sent to all Nexus BlackBerry users whose devices had not made contact for at least three months. The responses (or non-delivery reports!) from those people have allowed a further clean-up of inactive users.

Following this work the number of users remaining on Nexus’ BlackBerry Enterprise Server is now down to 33. A further four names look eligible for removal, since those users’ BlackBerries haven’t made any connection to our servers during February. For now, however, they remain listed as active users.

Maintaining two BlackBerry Enterprise Servers, and a SQL database,  for such a small number of users does not represent a good return on the effort required. The department would struggle to justify provision of a new service from which fewer than thirty University members would benefit.

Nexus’ BES service will therefore be retired ahead of the migration to Office 365. Current BlackBerry users should not replace old devices with new Blackberry handsets unless they propose to connect them to Nexus only via the ActiveSync protocol.

https://help.it.ox.ac.uk/nexus/mobile/blackberry

Posted in Uncategorized | Leave a comment

Office 365, viral tenancies, and side effects

All of the University’s domain names have to be added to Office 365 in order to use them with the service. This process has been hampered both by the sheer number of domains in use here and Microsoft’s decision to allow users to self-register for the service.

Self-registration is described in the documentation as making life easier for the administrator. And if all you’re worried about is ensuring that your users can access Office software as easily as possible then that aim is achieved. However domains with self-registered users become, in Microsoft’s parlance, a ‘viral tenancy’ – anarchic and uncontrolled. To bring them back under central control one must first prove ownership of the domain. This requires:

  • A valid email address within that domain
  • Following a link in a confirmatory email sent to you.
  • Filling in a registration form.
  • Skipping the ‘invite other users’ message.
  • Answering the ‘become the admin’ invitation and collecting a verification text string.
  • Adding the text string into DNS.
  • Cancelling the ‘provide admin contact details’ message.
  • Verifying that the string found in a DNS query matches the one they provided.

Only once all of that has been done can you login to Azure via Powershell and disable any further self-registrations from taking place:

Set-MsolCompanySettings -AllowAdHocSubscriptions $false

In an organisation with, say, two domain names this isn’t too onerous a task. But when you have several hundred domains the lack of an automated way to do this becomes a trifle wearing. Each of these reclaimed domains can then be added into the official tenancy but this too requires a DNS text string to be generated, added, and verified for each one. It is the manual and repetitive aspects of this process which are taking time but we hope to have all University domain names under the central Office 365 tenancy by the end of October 2016.

Unintended Consequences

This process has also produced some other unexpected side-effects – many Nexus users will have used their University email address for things like XBox Live or Visual Studio accounts. Once the domain name has been formally ‘claimed’ for use in Office 365 it is no longer available for ad-hoc registrations for these other services.

"You can't sign up here with a work or school email address. User a personal email..."

Sign-up denied

 

We therefore recommend that all Nexus users ensure that they always use a personal email address when registering for a non-University service.

 

Posted in Uncategorized | Leave a comment

RDP: “The connection has been lost.”

This message was appearing a little too often for my liking:

download

“The connection has been lost. Attempting to reconnect to your session… “

In many cases this problem is caused by a feature known as Auto-Tuning. This is supposed to continually adjust TCP/IP receive window size based on the network conditions at any given moment. But on a less-than-perfect network this can cause time-out issues.

The TCP receive window size is the maximum amount of incoming data that can be buffered at once on the receiving side of a connection. The sending host can send only that amount of data before waiting for an acknowledgement and then a “receive window” update from the receiving host.

The TCP/IP stack nowadays tunes itself with larger default window sizes than it used to. Instead of using a hard-coded default value, TCP automatically adjusts the window size – beneficial during bulk data transmission by reducing the number of segments sent with large amounts of data.

Auto-Tuning continually determines the optimal receive window size by measuring bandwidth delays and the application’s retrieve rate. In an ideal world this means excellent performance even with changing network conditions. In the real world this means occasional timeout errors…

This doesn’t just affect Remote Desktop – some older version of Outlook used to get error 0x800CCC0F (“The connection to the server was interrupted. If this problem continues, contact your server administrator or Internet service provider (ISP).”) for the same reason.

What can be done?

To tweak your auto-tune settings:

Run CMD as an administrator and enter this command:

netsh interface tcp set global autotuninglevel=restricted

If you later wish to reverse this setting:

netsh interface tcp set global autotuninglevel=normal

To see what you’ve got right now:

netsh interface tcp show global

The permited values for the AutoTuningLevel parameter are:

disabled
Sets the receive window at the default value.
highlyrestricted
Lets the receive window grow beyond the default value, but does so very conservatively.
restricted
Lets the receive window grow beyond the default value, but limits such growth in some scenarios.
normal
Lets the receive window grow to accommodate most scenarios.
experimental
Lets the receive window grow to accommodate extreme scenarios.

N.B.  The experimental value may decrease performance and should only be used for testing.

 

Posted in Uncategorized | Leave a comment

RIP Smartscreen

Microsoft have included spam-filtering in Exchange for many years, under the SmartScreen name. But from 1st November that will change: there’ll be no more updates and the feature won’t be included in new versions of Exchange Server.

Updates for SmartScreen client-side spam-filtering for Outlook on Windows will also be ending, although SmartScreen will remain current in Microsoft’s web browsers. Despite sharing the name the in-browser feature protects against malicious websites.

If you’re using SmartScreen now the existing definitions you have will still be there, and will still work, but there won’t be any more updates to them after 1st November.

The motivation for this seems to be twofold – the blunt instrument of applying a spam-confidence level to an email isn’t very effective, and Microsoft’s online protection offering does a far better job. Realtime filters can react faster to spoofed email, hijacked accounts, and user feedback. Coincidentally Exchange Online Protection is chargeable for on-premises users, perhaps as a further incentive to move to Office 365.

There is no net effect on Nexus from this change (Oxmail does the job of spam-scoring for the University) but, as many of you know, messages categorised as legitimate can still end up in ‘junk’ because of Outlook applying filters – it’s why we’ve always advised that users set spam filtering via the SCL and not via Outlook’s SmartScreen. The absence of future updates mean that client-side spam filtering will only become less effective over time.

In the longer term Nexus’ move to Office 365 should ensure that Exchange Online Protection has an opportunity to take over spam-filtering responsibilities.

Posted in Uncategorized | Leave a comment

The New York Times on email chains

This is some excellent advice on the subject of “When I’m Mistakenly Put on an Email Chain Should I Hit ‘Reply All’ Asking to Be Removed?”
https://twitter.com/bydanielvictor/status/771698039908CrWfRgTXgAEQHJE[1]

Posted in Uncategorized | Leave a comment

“Why does Nexus send me spam?”

We have had a spate of spam making it into Nexus mailboxes recently, provoking a flurry of discourse on the ITSS-Discuss mailing list. To be able to stop spam more effectively there are some checks that IT Support Staff can do with their users prior to logging support tickets.

Spam detection isn’t as simple as virus/malware detection: there isn’t the same definitive certainty one gets when a virus’ signature matches a message’s attachment. To conclusively determine if a message is spam requires an understanding of context and language that is currently still far in excess of a humble computer’s capabilities. There is also a personal aspect – one person’s spam message is another’s crucial information. So spam isn’t deleted, no matter how certain we can be that that’s what it is. Instead we allow you to personally set your own tolerance to spam. If you do this correctly it will determine what goes into ‘Junk’ and what goes into ‘Inbox’.

Firstly though, here’s what is going on with incoming email:

ON ARRIVAL AT THE UNIVERSITY
New messages arrive at OXMAIL and are scanned. Details of Oxmail’s processes can be found here: http://help.it.ox.ac.uk/email/scanning/index and here: https://help.it.ox.ac.uk/network/smtp/relay/index

OXMAIL then sends messages on to their destination, which might be Nexus, forwarded to a personal email address outside the University, or to a departmental server.

ON ARRIVAL AT NEXUS
The spam score assigned by OXMAIL is translated directly into a Spam Confidence Level, or SCL. Each asterisk in the X-Oxmail-Spam-Level header is counted: the total becomes that message’s SCL. So this example ‘X-Oxmail-Spam-Level: *********’ would have an SCL of 9.

Nexus is able to automatically move spam messages into the Junk folder of your mailbox before you see them, but only if you tell it to do so. This is a two-stage process:

  1. Turn on the spam filter in OWA. This step often gets overlooked. Without doing this step any setting you apply won’t take effect. Instructions for doing this can be found here: https://help.it.ox.ac.uk/email/filter/index.
  2. Select your level of tolerance to spam. This is done via the SelfReg tool (https://register.it.ox.ac.uk/self/nexus).
    The higher the SCL is that you can accept, the fewer messages will be moved into Junk. The options equate to the following:
    OFF: All messages will go into your Inbox.
    LOW: Messages with an SCL above 7 will be moved to Junk.
    MEDIUM: Messages with an SCL above 5 will be moved to Junk.
    HIGH: Messages with an SCL above 3 will be moved to Junk.

WHY DO YOU TELL US TO TURN OFF JUNK FILTERS IN OUTLOOK? (or other email clients?)

The assessment of spamminess done by OXMAIL is logged, consistent, and repeatable. If two people are sent the same message, and have the same spam-filtering preference set, then the same thing should happen to that message for both people.

End-users’ applications aren’t so clear-cut. Each program can use widely differing technologies to assess spam and the software which scans it isn’t always the same version, even when the application is the same. Furthermore, messages that are moved to Junk by the email client application can’t be differentiated between a user choosing to move a message between folders or their software doing it for them. All that Nexus’ servers see is an end-user request for a message to be moved.

In other words Outlook, and others, can do a pretty good job of spotting and filtering spam after it’s delivered. But we can’t guarantee that your program won’t falsely-identify a genuine message as spam. We can’t guarantee that your experience will match that of the person next to you (do they update as often as you do?). We can’t see in our logs what process decided to move that message between folders.

By deactivating client-side spam filtering we can be sure that the only spam processing that does happen is recorded in message headers.  This gives us a fighting chance of diagnosing any issues with the process. If you do spam filtering in your application you are muddying the waters for us if things misbehave and you want us to help you to troubleshoot.

WHAT CAN I DO BEFORE LOGGING A TICKET?

  1. Check that the user has set spam processing in SelfReg AND has turned on spam processing in OWA.
  2. Review the headers of an offending message to see what Spam Score it has been given.
  3. Verify if the user’s application software, or rules processing, or macros, or any other such process might be doing additional spam processing / filtering.
  4. Ensure that the end-user hasn’t configured a whitelist within their email application that is over-riding the spam-processing values they have set.
  5. Record the outcome of these tests in the details when you log a ticket for us to investigate why you are receiving spam to your inbox.

nexus_itss_spam-processing[1]

 

EDIT 1st August 2016:

In the case of a shared mailbox SelfReg does not currently allow you to alter the spam preference – it is limited to the currently-logged-on user. However the Nexus team can manually apply spam preference values for you if spam becomes an issue within a shared mailbox.

Posted in Uncategorized | Leave a comment