Over the course of the last week Readify has migrated from BPOS to Office 365. For most BPOS customers this should be a non event since Microsoft will work with you to drive the migration process and all you’ll have to do is look after the local client side issues – so normally I would recommend that you wait for Office 365 to come to you courtesy of the Office 365 migration teams that are now moving customers.
A Customer Driven Migration
Our situation was a little bit different however. We adopted BPOS quite early and in our rush to move onto it we didn’t stop to think about which AD domain we were going to associate with the environment. At the time we had two AD domains in use within Readify. We decided to associate with our old one since that is where most of the users were at that point in time. What we didn’t realise was that once associated you were really locked into using that domain unless you wanted to completely remove your user accounts and start the AD synchronisation again (note: I’ve heard that there is a way around this now, but it certainly wasn’t available when we first asked).
Anyway – so our migration was a little bit more complicated because not only did we want to move from BPOS to Office 365, but we also wanted to start synchronisation with a completely new AD domain which was now the home for user accounts, security groups etc.
In the end our migration wasn’t really a migration at all. We simply created a new Office 365 account, and used the MigrationWiz tool to copy mailbox content across from BPOS into freshly minted mailboxes at Office 365. The plan started simple but we hit a stumbling block which resulted in a full day e-mail outage, and no rollback path.
What cooked our goose?
Basically what happened was that there was a resource record in BPOS that was somehow incorrectly located inside the BPOS AD environment. This stopped us removing it, and the associated SMTP proxy address which still tied it to the @readify.net domain name. So even though we had disassociated the @readify.net address from all users, removed the @readify.net domain name from BPOS and requested the clean-up of the FOPE (Forefront) environment in Microsoft Online Services, we were still blocked from verifying the domain in Office 365.
We attempted to re-associate the @readify.net domain name in BPOS, however for some reason the e-mail routing rules in FOPE (presumably) didn’t get re-established properly so we ended up with bouncing e-mails – so our rollback was cut-off. After a conversation with the support team at Microsoft we decided the best course of action was to move forward by removing our BPOS environment completely. This meant that we had to step up the actual mailbox migration which we were hoping to do in the background for users whilst Office 365 started to receive new e-mails.
The MigrationWiz tool is good, but it can’t alter the laws of physics, there was only so much data we could pull out of the BPOS environment. Interestingly transfer into Office 365 was several times faster than transfer out of BPOS so we suspect that there was some throttling going on there. We also had a few users with big mailboxes (mea culpa).
A Devine Hack
Eventually we got all the data across and we gave Microsoft the green-light to decommission the BPOS environment. Unfortunately that item sat in the queue for way too long and we were looking at our second day of an e-mail outage. That was when our system administrator, Nathan, had the bright idea of setting up an external e-mail relay that would translate @readify.net addresses to the @readify.onmicrosoft.com addresses that get associated with an Office 365 instance automatically. This largely worked although a few of our external facing DLs weren’t mapped so we weren’t running at 100%, but at this stage I was happy with 50% Outbound e-mails also went out with @readify.onmicrosoft.com and later @readify.net.au (which we had successfully verified earlier).
The Plea for Help
This was the status quo for most of last week but the job wasn’t finished so I got back onto Microsoft’s case about cleaning up our BPOS environment. After a fairly disappointing call to Microsoft support I put this tweet out to the Internet. That certainly stirred up a bit of a hornets nest. Its not that I have any particular sway within Microsoft but I’m usually pretty pro Microsoft (I like them as a company), it was unlike me which is why I think it got some attention.
Simultaneously we also reached out to some other contacts within Microsoft who were associated with Office 365 and asked for help and this is really what got the ball rolling for us. We got a local contact who managed to find someone in the US who could look directly at the problem for us rather than going through 2-3 layers of support with no real access. Once we got access to these people it took less than 36 hours to nail the problem which had been blocking us for around five days.
Suggestion for Improvement
I think that if there was one improvement that I hope Microsoft would make to Microsoft Online Services, especially BPOS/Office 365 it would be to make it possible to get access to these people without a series of secret handshakes. What took over 20 phone calls to get no where was resolved with 2-3 phone calls to the right individuals. Customers and partners need a panic button of sorts.
On the other hand Windows Azure support has been consistently good, when you make a request they usually know what you are talking about and can help you when you have problems. When you think about the products involved however that makes a lot of sense. BPOS/Office 365 sits on top of Exchange, SharePoint, and Lync – all of which have very specific requirements around DNS/AD – whereas Windows Azure is somewhat independent of all that, at least for the core services.
Credit Where Credit Due
At the height of my disappointment with the experience we were getting I put out that tweet I mentioned earlier. After that the troops within Microsoft did rally to get our problem solved. Specifically I’d like to thank the following people at Microsoft (in no particular order):
- Steven Wilks
- Andrew Pasco
- Lee Hickin
- Vajira Weerasekera
They took the time to look into the issue and respond to some e-mails, take phone calls which makes a huge difference.
Internally within the Readify support team, last week is already being referred to as “hell week”. I hope I never have to go through anything like that again. But we now know a lot more about the way the internal support organisation at Microsoft works and the importance of finding someone who understands the problem and can show some empathy. It taught me that Readify’s own application support offering has to have as few layers as possible between the engineers and the customers.
I’d like to make a big shout out to a few of the Readify guys, including Andrew Harcourt, Tatham Oddie, Nathan Thomas, Robert McCann, and Tien Phan – all of who spent time after hours working through the problem. Hopefully I haven’t missed anyone.