BlackBerry Enterprise server versions

Mostly for my own reference here’s how the BlackBerry bundle numbers you see in ‘Add/Remove programs’ correspond to the various maintenance releases and hotfixes:

Updated 13th April 2014

Version Bundle
5.0 223
5.0 MR1 236
5.0 MR2 244
5.0 MR3 255
5.0 MR4 267
5.0.1 (Service Pack 1) 70
5.0.1 MR1 82
5.0.1 MR2 117
5.0.1 MR3 139
5.0.2 (Service Pack 2) 36
5.0.2 MR1 51
5.0.2 MR2 96
5.0.2 MR3 119
5.0.2 MR4 133
5.0.2 MR5 146
5.0.3 (Service Pack 3) 33
5.0.3 33
5.0.3 MR1 41
5.0.3 MR2 53
5.0.3 MR3 93
5.0.3 MR4 107
5.0.3 MR5 143
5.0.3 MR6 163
5.0.3 MR7 227
5.0.3 MR8 256
5.0.4 (Service Pack 4) 38
5.0.4 MR1 52
5.0.4 MR2 70
5.0.4 MR3 86
5.0.4 MR4 100
5.0.4 MR5 116
5.0.4 MR6 128
5.0.4 MR7 160

Service Pack 4 Maintenance Release 7 is dated 9th April 2014. The minimum requirement to install this remains the same: the server must be running at least v 5.0.4 (bundle 38).

Posted in Uncategorized | Leave a comment

iOS6 devices hijack ownership of an Exchange meeting

It has been brought to our attention that Apple devices which have been upgraded to iOS6 may encounter an issue when responding to meeting requests in Exchange. The behaviour has been described as ‘meeting hijacking’ and seems to relate to situations where a user opens a meeting request in Outlook but takes action on it from iOS6.

The effect is that, under certain circumstances, taking action on a meeting request (such as accepting a meeting) has the unfortunate side-effect of simultaneously taking ownership of that meeting. Under those circumstances the meeting updates, cancellations and confirmations would be directed to the new owner rather than the original organiser of that meeting.

Both Apple and Microsoft are reported to be investigating this problem and, at the time of writing, the advice is to contact Apple for an update.

Microsoft have documented this issue here: http://support.microsoft.com/kb/2768774.
The Apple discussion forums discuss it here: https://discussions.apple.com/thread/4368289?tstart=30

This issue appears to be confined to those devices which run iOS6 AND have been configured to connect to Exchange using ActiveSync. If you use a different protocol (such as IMAP4 or POP3), you have a non-Apple device, or you’ve not upgraded your iDevice to iOS6 you should be unaffected.

UPDATES:

The Exchange Team Blog reports on the issue here: http://blogs.technet.com/b/exchange/archive/2012/10/23/ios6-devices-erroneously-take-ownership-of-meetings.aspx

Michael Rose’s blog also documents this behaviour as an even longer-running issue. His article describes a user who reproduced the problem:

At some point, the iOS device syncs the calendar via ActiveSync and suddenly becomes confused about who the owner of the meeting should be (the organizer, in Exchange-speak). The iPhone decides that its owner should become the organizer, since it has no idea who the real owner is, and syncs this property change back to the Exchange server. Exchange 2007 now has a disconnected copy of the meeting with a different owner. Exchange is agnostic about this.
Now the iPhone owner declines the meeting for whatever reason. Exchange automatically generates a cancellation or decline notice and sends it out to everyone since the disconnected copy of the meeting has a different owner. This results in mass confusion and sometimes will delete the meeting from the other calendars.
We verified this problem against iOS 4, 5 and 6 with Exchange 2007 and 2010. In Exchange 2010, Microsoft introduced a “calendar repair agent” that is supposed to detect this problem and resolve it. This calendar repair agent is a daily timer job. Microsoft did release patches on Exchange 2007 SP2 and up to correct some of the issues that are similar to this, but this particular problem was never resolved.

It seems that blame can be applied to both parties in this. Michael’s user’s analysis suggests that:

 Apple’s manipulation of the organizer field is against the ActiveSync specification. However, ActiveSync will not stop iOS from doing this regardless of the fact that it is “against the specification.” ActiveSync will happily accept the change and write the properties from the mobile device even if the ActiveSync spec says that Exchange explicitly should not do this.

Based on this it reads to me that blame should be apportioned in both directions:

  •  Apple: wrote code that attempts to do something it shouldn’t.
  • Microsoft: wrote code that accepts a change that it shouldn’t.

Now we just need one of them to take ownership and fix the issue.

 

Posted in Uncategorized | Leave a comment

Exchange 2010 SP2 Rollup 4 doesn’t like a wildcard in your ‘Accepted Domains’

We have identified a curious issue with Exchange 2010 when SP2 Rollup 4 is applied. Here’s what happened, in a nutshell, on our servers:

 Users who have ‘Full Access’ permissions over a secondary mailbox lose the ability to open that secondary mailbox via OWA.

The problem was clearly directly attributable to the rollup and we confirmed this: prior to SP2 RU4 it works; after the update is applied it no longer works. Removing rollup 4 restored the functionality and stopped the error (‘System.ArgumentNullException’) from being returned.

Initially we were confused both by this behaviour and also that it didn’t apparently seem to be a widespread issue. Had we found a problem that was unique to our installation? After our diagnostics had drawn a blank we naturally escalated our report of the problem. A bit of email to-ing and fro-ing eventually led to confirmation that Microsoft can reproduce the error we’re experiencing – but only under a certain very specific condition. This seems to be the explanation for why the issue isn’t widespread – our installation differs in one crucial respect from the majority of others out there. This problem is caused by a revision, introduced in SP2 RU4, which changes the server’s behaviour when trying to access secondary mailboxes from within OWA.  Here is what we’ve been told:

“When there is an additional mailbox that is being accessed through OWA, we perform a search for the same and return it back based on the permissions on the mailbox. The way this search is being performed has changed starting E14 SP2 RU4.

In Exchange 14 SP2 RU3 and previous versions, we just did an un-scoped search. So in a large organization and in a hybrid setup (cloud + On Premise), the search results could be ambiguous. There was a change in this logic introduced starting Exchange 14 SP2 RU4. We now do a scoped search, using the scope of the Primary SMTP Domain of the additional mailbox. We go ahead and search for an Organization ID for this domain in the OrgID Cache that is built. This Cache is built based upon the list of domains in the “Accepted Domain List”. When this domain entry is not in the “Accepted Domain List”, the cache would return a null organization id, resulting in the System.ArgumentNullException. We need to also note here that the cache is built upon the actual domain name, and not a wild card. So a domain entry in the Accepted Domain List, similar to “*.something.com” does not mean that we will have all the domains under something.com in the cache. We will need to add all the domains explicitly in the Accepted Domain List. The cache will be formed based on this list, so that we need not iterate the accepted domain list frequently.”

The university, by virtue of its many departments, divisions and colleges has a great many domain names – 208 at the current count* – and this list changes regularly. To avoid the maintenance overhead of altering the Accepted Domain List we have long since used a wildcard entry for the ox.ac.uk domain. It seems that we therefore fall foul of this new scoped search functionality as a consequence of our wildcard. At the moment the ‘fix’ seems to be to remove the wildcard entry and instead add an entry for every accepted domain.

*Some way off the limit.

Posted in Uncategorized | Leave a comment

Unintelligent Intelligent Provisioning

This month brings us some new hardware which will shortly become our new Threat Management Gateway array. The new servers are all HP’s  Proliant DL380p (Gen8).

The process for setting up new HP servers has recently been updated. They’ve recently integrated their set-up process internally to the server: instead of booting from a SmartStart CD there now is a built-in ‘Intelligent Provisioning’ boot option instead. In fact a Gen8 server refuses to boot from a Smartstart CD any more.

This new Intelligent Provisioning  feature is accessed by pressing F10 at the crucial moment at startup.  For six of our new servers this option was both quick and superbly simple – IP’s options start with a quick config to specify performance options, the ability to update system software as part of the deployment and of course to configure the disks using the Array Configuration Utility (ACU). When accessed via iLO it also allows an ISO file to be presented as virtual media for the installation, so this was my preferred route to do a deployment.

All was well for the first batch of new servers – the array was set to be customised at this first stage, so the ACU launches, lets you configure the disks and then resumes the installation. However this process wasn’t quite so seamless for the last two servers. These two were due to become our new SQL mirror and, initially, all seemed comfortingly familiar.

However beneath these three familiar options the ‘continue’ button had disappeared. And almost immediately an error message appeared on screen:

“There are no physical disks attached – you will need to attach a supported drive to continue”

Now I knew that these disks were there. I’d seen them. And this same error was appearing on two different servers which were installed in two different datacentres. Neither had anything more connected to them at this stage than a power feed and a single network cable for the iLO connection – not even so much as a keyboard and screen. But paranoia had started to set in. It didn’t take long for my diagnostic approach to reach the point where I was thinking about removing the cover from the server to check internal cables and re-seat hot-swappable drives.

But prior to that a quick review of this error on the net seemed to be in order. Search engines weren’t very forthcoming, usually all seemingly directed at one or more variants of the same PDF (it suggests that the issue relates to an SD card or an unsupported drive being connected). Neither of these applied in our case.

The fix is surprisingly straightforward however. Rather than request an array configuration as part of the intelligent provisioning process, instead launch the ACU separately. Configure the disks as you wish in advance, and then return to Intelligent Provisioning afterwards. This step seems to bypass whatever disk-hiding glitch was present in the system and setup then resumes as advertised.

Posted in Uncategorized | 2 Comments

Mobiles revisited

It’s been some time since I last wrote about Nexus’ mobile users so here’s some nice fresh statistics to see what has changed:

  •  11,461 ActiveSync devices
  • 78% have connected in the last three calendar months.
  • 61%  have made a connection today.
  • 3,031 users have more than one ActiveSync device

 

Effectively we have just under a quarter of our users making regular connections via ActiveSync devices. Although Apple have the, er, Lion‘s share and are showing a healthy increase, the gong for greatest percentage increase goes elsewhere: the number of Samsung devices has more than tripled. About half of them are the Galaxy S2 and 43 people are using the Galaxy S3.

 It’s also worth noting that the Windows Phone is now starting to appear in respectable numbers and is also showing healthy growth.

And finally, a few other numbers that caught my eye:

  • Motorola (33 devices)
  • LG  (10 devices)
  • Research In Motion’s Playbook (10 devices)
  • Palm (22 devices)
Posted in Uncategorized | Leave a comment

Using the New-MailboxRepairRequest cmdlet

This cmdlet was mentioned in a previous blog post but I’ve noticed that the information on it that’s out there can be a bit sketchy. So, for my own reference as much as for anyone else’s, here’s my notes on it:

New-MailboxRepairRequest can be run against a whole database (like its predecessor, ISINTEG) or against just one mailbox within it. If a repair is run against a single mailbox, only that mailbox will have service interrupted: all other users within that database are unaffected.

There are four areas that can be checked:
  •  Search folder corruption
    This option looks for all folders named in ptagSearchBacklinks, ptagSearchFIDs, and ptagRecursiveSearchFIDs. It verifies that each folder exists. If the folder no longer exists then it removes that folder from that list.
  • Aggregate counts on folders
    Tallies up all the messages in a folder and keeps a running total of counts and sizes. Once the iteration is complete it will verify the computed counts against the persisted counts on the Folders table record. If there is a discrepancy it will update the persisted counts with those it has calculated.
  • Provisioned folders
    Checks for provisioned folders with unprovisioned parent folders or vice versa
  • Folder view
    Iterates all views for a folder then reconstructs a temporary copy of them. If there is a discrepancy between the two it will delete the view so it can be rebuilt from scratch the next time it is requested.
This cmdlet also includes a detectonly switch, if required, to simply report on problems without making changes. This switch doesn’t seem to affect user service (when tested). It should be safe to use even when a user hasn’t been notified of a service interruption. However that point may be moot: to repair any detected damage with this cmdlet you will affect the user.
g

Examples

A check on a user’s mailbox’s folder views, but without undertaking a repair, would be similar to:

New-MailboxRepairRequest -Mailbox <MailboxID> -CorruptionType FolderView -DetectOnly

The ‘MailboxID’ value can be a GUID, DN, UPN, LegacyExchangeDN, SMTP address, alias or in the format ‘domain\user’.
A more thorough check of a user’s mailbox, reviewing all four checkable areas at once, and completing a repair, would interrupt the user’s service. The command would look like this:
New-MailboxRepairRequest -Mailbox <MailboxID> -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView
A check on a database, for search folder corruption only, (and repairing any errors found) would be similar to this:
New-MailboxRepairRequest -Database <DatabaseName> -CorruptionType SearchFolder

Output

There is no direct output from this tool into the Powershell console. To see what’s been found you must open the application event log of the Exchange Server which hosts the mailbox (you may need to check which is the active database) . Start by looking for MSExchangeIS Mailbox Store events with the event ID 10047 and 10048. To make things a little more challenging note that if you’ve run the New-MailboxRepairRequest cmdlet more than once the event log will only show the mailbox by GUID.  To assist in finding the right one you may therefore want to run Get-Mailbox <name> |FL name,ExchangeGuid.

Event ID

Description

10044

The mailbox repair request failed for provisioned folders. This event ID is created in conjunction with event ID 10049.

10045

The database repair request failed for provisioned folders. This event ID is created in conjunction with event ID 10049.

10046

The provisioned folders repair request completed successfully.

10047

A mailbox-level repair request started.

10048

The mailbox or database repair request completed successfully.

10049

The mailbox or database repair request failed because Exchange encountered a problem with the database or another task is running against the database. (Fix for this is ESEUTIL then contact Microsoft Product Support Services)

10050

The database repair request couldn’t run against the database because the database doesn’t support the corruption types specified in the command. This issue can occur when you run the command from a server that’s running a later version of Exchange than the database you’re scanning.

10051

The database repair request was cancelled because the database was dismounted.

10059

A database-level repair request started.

10062

Corruption was detected. View the repair log details to see what the corruption type was and if it was repaired.

To make these events easier to find, you may want to create a custom view in the Event Viewer:

  1. On the Action menu, click Create Custom View.
  2. In Create Custom View, click By source, and then in the Event sources list select MSExchangeIS Mailbox Store.
  3. In the box labelled <All Event IDs>, add the event IDs for the repair request events that you want to see. For all of this cmdlet’s events enter 10044,10045,01146,10047,10048,10049,10050,10051,10059,10062.
  4. Click OK.
  5. In Save Filter to Custom View, type a name for this view.
  6. Click OK.
N.B.

To ensure that performance isn’t negatively impacted by this tool it is limited to working on one store at a time per server, or to examining 100 mailboxes.
This tool has a partner utility for public folder databases (New-PublicFolderDatabaseRepairRequest) which will accept only ReplState as the corruption type to query. All other syntax is the same.

Posted in Uncategorized | 1 Comment

Mission accomplished!

To save you reading all those previous posts, here’s a recap for you:

Late in the evening of Sunday 4th March 2012, the testing phase was over. I had had feedback from our early adopters and, with all systems set to ‘go’, the first batch of production users were about to be migrated from Exchange 2007 into the heady delights of Exchange 2010. Over the course of that night these accounts tested the efficacy of my scripts and logging – which actually performed better than expected. Script processing was far faster than I could have dared to imagine.

The migrations continued each night, moving approximately 2500 mailboxes each night, for five nights each week.
By Monday 19th March, after three weeks’ worth of migrations, I had passed the halfway mark. The three-quarter mark was passed by the 23rd March and by the time I reached the end of the month over 99% of Nexus mailboxes had been successfully migrated.

April began with negotiation: within the remaining 306 Exchange 2007 mailboxes were 150 mailboxes belonging to one particular division which had been postponed due to sharing concerns. It took until the 16th April to resolve their issues before those mailboxes could finally be migrated. This took me up to the grand total of 99.8% completed. But at this stage I was entering the hard slog of problem mailboxes – the ones which had already failed to migrate at least once. The reasons for this were quite varied, starting from the ones which had (as it turned out) relatively straightforward corrupt messages, through to a pair of mailboxes where every migration attempt locked the user out of their mailbox for 24 hours. Resolving these last 150-odd mailboxes involved a significant amount of communications both with users and their IT support staff to work through the many lists of corrupt mailbox items. I owe a debt of gratitude to my colleagues – and to the users themselves -  for their assistance with this part of the work.

At the same time I had to begin recovering space from the Exchange 2007 servers by defragmenting databases. Migrating users from Exchange 2007 had created vast swathes of whitespace within that system’s stores so our backup software still saw the databases as enormous. In order to ensure that both versions of Exchange could still be backed up successfully in the limited time available each night a defrag was the obvious solution. After my initial (and very time-consuming) manual approach to this I developed a script to dismount the empty databases, defragment them with ESEUTIL and then remount them. The sole remaining manual step was to kick off a full backup so that our schedule of full and incremental overnight backups didn’t get confused.

My inability to successfully move the final three users required vendor assistance but, after a number of dead ends, and time pressure looming, a mailbox backup and restore emerged as the only successful way to migrate them across. The last of these users was migrated in this way this afternoon.

I therefore pronounce that, as of 2:48pm this afternoon, Exchange 2007 is officially no longer servicing any production Nexus mailboxes.

Let the decommissioning commence!

Posted in Uncategorized | 2 Comments

We’ve just been Street Viewed…

 

In the spirit of The Register‘s mission to snap the snappers here’s my contribution. As one of Google’s cars passed the office this morning I took the opportunity to do unto Google as they were doing unto me:

EDIT:

Sure enough, I can see myself in the first-floor window snapping my picSnapping the snappers:

 

 

 

 

 

 

Posted in Uncategorized | Leave a comment

Mailboxes that just won’t migrate…

With over 48,500 mailboxes now running successfully on Exchange 2010 it’s tempting to think of the migration as ‘done’. But appearances can be deceiving: at the beginning of last week I was stuck with twelve mailboxes remaining –  none of which would migrate. Because I’ve been using the ‘SuspendWhenReadyToComplete‘ switch it allows the job to get to the autosuspended stage successfully. That gives an illusion of success – but any attempt to actually complete the migration of these mailboxes failed, at the final hurdle, with a MAPI error.

Initially there didn’t seem to be a great deal of obvious help when reviewing the migration logs either. The logs reported zero corrupt items so the usual fix (deleting the problem messages) couldn’t apply. What you do see is a failure type listed as:

MapiExceptionInvalidParameter

This is elaborated a little as:

 Error: MapiExceptionInvalidParameter: Unable to modify table. (hr=0×80070057, ec=-2147024809)

Sadly it transpires that this is far from an unusual reason for a move to fail – there are literally dozens of posts and queries about it on the search engines – and the calibre of the options you’re presented with to fix it vary wildly depending on which sites feature most prominently in your preferred engine’s results.

Let’s take a look at what’s recommended.

  • Rules and Out-Of-Office
    The standard recommendations usually start with a suggestion that the user searches for (and removes) their rules from the mailbox, also checking for broken Out-Of-Office messages (which are often overlooked although they’re also a kind of rule). To add insult to injury hardly anyone ever seems to mention that Outlook has the facility to export those rules beforehand. I’d be pretty cheesed off it all of my carefully-crafted rules were callously wiped out without that simple prior precaution:So what’s the next step? Once you’ve got that backup of those rules (!) conventional advice is to  launch Outlook with the /CLEANRULES switch to make sure that you really are rid of them.  But that’s not terribly helpful if that user’s only email client is on Linux.
  • MFCMAPI
    Many sites recommend delving into the mailbox using a tool such as MFCMAPI. Our strict rules about user mailbox privacy seemed pretty certain to preclude me from going down that route except as a last resort.
  • Export-Mailbox
     This Powershell cmdlet is the server-side equivalent to an Outlook export but it has a far lower tolerance to MAPI errors. It was unlikely to work here and, even after approval to try it for one user had been obtained, it bombed out with MAPI errors, as expected.

The most viable remaining option for me – to avoid any accusation of privacy-violation – seemed to be to reattempt the move invoking the -IgnoreRuleLimitErrors switch but, again, that was running the risk of jeopardising user content. I’m sure it’s all a lot easier in an organisation with managed desktops rather than our ISP-style service! Disappointingly it seemed that, in cases like these, there was always going to be some kind of imposed data loss but with a likelihood of not knowing exactly what was being sacrificed.

Instead I returned to the logs looking for inspiration from a couple of these mailboxes that I had managed to migrate. Below is an abridged version of what a failed-with-MAPI-error migration log looks like. The vital clue is right at the end:

18/05/2012 14:10:34 [MBX09] ‘<name>‘ created move request.
18/05/2012 14:10:37 [CAS03] The Microsoft Exchange Mailbox Replication service ‘CAS03.ad.oak.ox.ac.uk’ (14.1.355.1 caps:07) is examining the request.
18/05/2012 14:10:37 [CAS03] Connected to target mailbox ‘Primary’, database ‘db005′, Mailbox server ‘MBX01.ad.oak.ox.ac.uk’ Version 14.1 (Build 218.0).
18/05/2012 14:10:37 [CAS03] Connected to source mailbox ‘Primary’, database ‘EXMBX04\SG03\EXMBX04SG03′, Mailbox server ‘EXMBX04.ad.oak.ox.ac.uk’ Version 8.3 (Build 192.0).
18/05/2012 14:10:37 [CAS03] Request processing started.
18/05/2012 14:10:37 [CAS03] Mailbox signature will not be preserved for mailbox ‘Primary’. Outlook clients will need to restart to access the moved mailbox.
18/05/2012 14:10:37 [CAS03] Source Mailbox information before the move:
Regular Items: 25508, 2.545 GB (2,732,488,589 bytes)
Regular Deleted Items: 670, 55.41 MB (58,104,875 bytes)
FAI Items: 27, 0 B (0 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
<SNIP>
18/05/2012 14:19:09 [CAS03] Final sync has started.
18/05/2012 14:19:19 [CAS03] Changes reported in source ‘Primary’: 0 changed folders, 0 deleted folders, 1 changed messages.
18/05/2012 14:19:19 [CAS03] Incremental Sync ‘Primary’ completed: 1 changed items.
18/05/2012 14:19:22 [CAS03] Fatal error MapiExceptionInvalidParameter has occurred.
Error details: MapiExceptionInvalidParameter: Unable to modify table. (hr=0×80070057, ec=-2147024809)
Diagnostic context:
  <SNIP>
——–
Folder: ‘/Top of Information Store/Deleted Messages’, entryId [len=46, data=<long string of hex>], parentId [len=46, data= <long string of hex> ]
18/05/2012 14:19:22 [CAS03] Relinquishing job. 

There’s the problem! A folder called ‘Deleted Messages’ which is found right at the top of the Information Store for eight of the remaining users’ mailboxes. This is beyond a coincidence and strongly suggests to me that this folder has been added by a particular email client. I have my suspicions about which one…

Problem solved!

Sure enough, if the user removes this ‘Deleted Messages’ folder from their mailbox they can then be migrated successfully.  It seems that it’s not the messages within it that are the problem – it all seems to be tied in to that particular folder, empty or otherwise. A review of the failed migration logs for the other remaining users found a similar issue for two of them: a folder called ‘Trash’. The similarity of likely purpose suggests that a similar client-based trend could be at work here. Again, I’m hoping that the users will remove these folders and the migration will succeed.

This still leaves us with two unfortunate users who have a different problem with their mailbox migration: the move hangs at the ‘completing’ stage. This is the one point where a user is actually blocked from their mailbox and in Ex2010 it’s rarely noticeable (it should be momentary). But with these two users the ‘completing’ stage keeps on going. Eventually it times out and permits the user back into the mailbox that it’s been blocking. I’ve had to tweak the MSExchangeMailboxReplication.exe.config file again (on a default installation you’ll find it here: C:\Program Files\Microsoft\Exchange Server\V14\Bin) to bring the MaxRetries property down from its default of 60. Without that change these two poor users would effectively be blocked from their email for the best part of a day every time I reattempted their upgrade.

 Last Resort

If all else fails it’s likely that you’ll need to bite the bullet and dive into the world of database repair tools. Backups, Outlook exports and taking a copy of the EDB file are all sensible precautions before going any further.

The old favourite, ISINTEG, makes what will probably be its last ever appearance here. I’m unlikely to need it again in the future: it’s been superseded by two Powershell cmdlets (New-MailboxRepairRequest  and New-PublicFolderDatabaseRepairRequest) for Exchange 2010. I won’t mourn the loss of ISINTEG much though. The new cmdlets don’t require the database to be dismounted and can be limited in scope to just the one mailbox under examination which represents a huge leap forward. It’ll also be nice not to have to leave Powershell and head over to the command prompt just to get it to work. For old times’ sake, here’s the syntax:

Isinteg –s /<servername> –fix –test alltests

The final step could well be the nuclear option of ESEUTIL. Aside from one relatively harmless switch (/D for defragmentation)  all of the likely uses for this tool will lose something from the store. Rather than go there though in this instance I’m able to escalate the issue. So I’m awaiting Microsoft’s recommended solution rather than ploughing on alone. I’ll update this post when they’ve provided some further ideas.

 UPDATE 28th May 2012: The Last Three Users

The last three Exchange 2007 mailboxes included the two mailboxes which had the hang-on-commit delay and a third where the corruption was the ‘Deleted Items’ folder itself. After several false starts and dead-ends (including an attempt to remove Deleted Items using MFCMAPI and use of Outlook’s /resetfolders switch) I eventually had to concede that time was against me.  It would have been preferable to both understand and solve the issue but to continue diagnosing would potentially leave these users’ mailboxes without backup, due to an imposed end of May deadline for ceasing the Exchange 2007 backup service.

The workaround was to export the mailbox content, rules (and every conceivable account setting) then disable the mailbox and create a brand-new one on an Exchange 2010 server. The settings were then reinstated (so the user could get back to sending and receiving) while the mailbox’s backup was gradually restored, backfilling the mailbox. The orphaned Exchange 2007 mailbox was then connected to a generic created-for-the-purpose user just in case any of the mailbox content hadn’t made it across.

Posted in Uncategorized | 5 Comments

Batch migrations

To keep our migrations under control I’ve been relying very heavily on the use of batched migration runs.  The process I’ve followed uses the ‘suspendwhenreadytocomplete’ value which performs the normal migration, copying the entire set of users’ mailboxes, but – crucially – stops the process just before the final stage where the changes are committed to the directory.

The really great thing about doing things this way is that, almost as an incidental benefit, the vast majority of corrupt mailbox content will be revealed through this exercise: those mailboxes fail to reach that ‘autosuspended’ stage. Yet getting to that point doesn’t  impact end users at all: they remain active and oblivious on their old server all the way through the job.

I used this feature to benchmark the migration process for one department and get a feel for for how long it might take to do this on a larger scale. When the final stage was ready we could just resume the task, via an overnight scheduled task. The momentary interruption when the mailbox is locked, for that final ‘commit’ phase, shouldn’t affect anyone. Our tests showed that even users actively accessing their mailboxes during the move weren’t adversely affected. Well, except for Mac users… More on that later.

Initially we had planned to use this auto-suspend capability to migrate the entire user base of 50,000 mailboxes in one go, but a lack of documentation from anyone else having tried it on that scale caused some raised eyebrows. The compromise required a rethink on mailbox distribution and some careful tweaking of circular logging. I used circular logging to keep log files manageable during the phase where mailboxes get copied. This was followed by a full backup, then circular logging was switched off before finally committing the changes. Although this added extra steps it did ensure that our backups were able to cope with the extra content without contending with vast numbers of logfiles that would’ve otherwise been generated by the mailbox moves.

There were several distinct phases to our migration which, due to our scale, were to be repeated across twenty separate migration runs. Each one went through these steps:

  1. Export from the GAL a CSV file containing the aliases of the mailboxes to be upgraded. I included primary SMTP address and department data so that a version of this same file could be used for a mail merge in the next step.
  2. Notify users of the upgrade – one week prior to the planned date.
  3. Create the move requests and launch them, using the ‘SuspendWhenReadyToComplete’ option.
  4. Turn off circular logging on the destination databases and back them up.
  5. Schedule an overnight ‘resume’ of the autosuspended mailboxes to complete the migration.

I came up with a bit of PowerShell (later optimised a little further by a colleague by the addition of logging and setting file attributes) to allow the bulk of the mailbox copying to take place without colleagues needing to remember the exact syntax of the command. It relies on you having a prepared CSV file with at least the following values in it:

Alias,TargetDB
UserAlias1,Database1
UserAlias2,Database2

The script assumes you’ll have saved the file as ‘c:\MIGRATION.CSV’.

All you need to do then is run the following as a PS1 file, remembering to include a name for the batch at the end:

param
(
$batchName = $(throw “Please specify a batch name.”),
$migrationCsv = ‘c:\migration.csv’
)

## Capture the batch name in a file (for the other scripts):
$batchNameFile = ‘c:\batchname.txt’
# If it already exists make it writable
if (Test-Path $batchNameFile)
{
Set-ItemProperty $batchNameFile -name isreadonly -value $false
}
# Overwrite the file (if it exists) and means the batch name is all it contains:
$batchName > $batchNameFile
# Make it read-only
Set-ItemProperty $batchNameFile -name isreadonly -value $true

#Load snap-in to support use of Exchange Commands:
Add-PSSnapin Microsoft.Exchange.Management.Powershell.E2010 -erroraction silentlyContinue
import-csv $migrationCsv |foreach {get-mailbox $_.alias |new-moverequest -suspendwhenreadytocomplete -batchname $batchName -targetdatabase $_.targetdb}

The job of completing these part-finished moves was left for an overnight scheduled task. To ensure that it only completed the moves for that night’s batch of users it would use the batch name that was created earlier:

#Load snap-in to support use of Exchange Commands:
Add-PSSnapin Microsoft.Exchange.Management.Powershell.E2010 -erroraction silentlyContinue
# Get the batch name
$batchNameFile = ‘c:\batchname.txt’
$batchName = Get-Content $batchNameFile
$TIMESTAMP_SUFFIX = “{0:dd-MMM-yyyy-HHmm}” -f (Get-Date)
$logFile = “C:\PS_LOGS\commit_$batchName_$TIMESTAMP_SUFFIX.txt”

“This script started executing at {0:G}.” -f (Get-Date) >> $logFile
“About to start processing the commits for batch: ‘$batchname’.” >> $logFile
## Resume and commit
Get-MoveRequest -resultsize unlimited -MoveStatus ‘AutoSuspended’ -BatchName $batchName | Resume-MoveRequest
#Resume any other suspended moves associated with this batch name:
Get-MoveRequest -resultsize unlimited -MoveStatus ‘Suspended’ -BatchName $batchName | Resume-MoveRequest

‘(Check the other logs and e-mails for precise timings & statistics).’ >> $logFile
“This script exited at {0:G}.” -f (Get-Date) >> $logFile

That took care of the heavy lifting but obviously I wanted to know what had happened when I arrived the next day, so yet another bit of scheduled PowerShell ran the following command and emailed me the output:

Get-MoveRequest -BatchName $batchName -MoveStatus Completed | Get-MoveRequestStatistics |ft alias, TotalItemSize, TotalMailboxItemCount, PercentComplete, BytesTransferred, ItemsTransferred -auto

In fact I ran several variations on that, with different status values, so I’d also be told about failed migrations and anything that was still suspended.

Now this is all well and good but we’ve now done nine nights of mass-migrations, as well as several early adopter and test ones, so after a while it’s possible that you’d lose track of all of those batch names you’d used. The problem is exacerbated because the graphical interface doesn’t even show them.

Luckily there’s another bit of PowerShell which can reveal what batch names are still lurking on your system:

Get-MoveRequest –ResultSize Unlimited | Sort-Object –Property batchname | Select batchname | Get-Unique –AsString

We’re about to cross the milestone of the halfway stage – we’ll have migrated approximately 25,000 mailboxes at some point in the early hours of tomorrow morning – so from that moment over half of the university will be running on Exchange 2010.

Posted in Uncategorized | 4 Comments