Exchange Server corruption and recovery

Yesterday, an unfortunate thing happened. After a reboot of our server (that was combined with the installation of some Windows updates), our Exchange Information Store became corrupt.

(Note: In our company, we use Exchange for internal company emails. All emails that go to our backupassist.com email addresses are hosted externally, so thankfully this Exchange corruption didn’t affect our helpdesk and tech support team.)

To compound matters, our staff member who is also our system administrator has been sick for nearly 3 months, so it was left to me (a developer!) to get our Exchange Server going again.

It took a few attempts, but thankfully having a good backup (made by BackupAssist!) saved the day.

The symptoms were these:

Every minute, this event is logged:

– Event ID 9175, Source MSExchangeSA, Category MAPI Session, Message: The MAPI call ‘OpenMsgStore’ failed with the following error:
The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server computer is down for maintenance.

When the Exchange Information Store service is started, this is logged:

– Event ID 9518, Source MSExchangeIS, Gategory General, Message: Error Read verification error starting Storage Group /DC=local/DC=CortexIT/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=CORTEXIT/CN=Administrative Groups/CN=first administrative group/CN=Servers/CN=GERSHWIN/CN=InformationStore/CN=First Storage Group on the Microsoft Exchange Information Store.
Storage Group – Initialization of Jet failed.
– Event ID 419, Source ESE, Category: Logging/Recovery, Message: Information Store (1220) First Storage Group: Unable to read page 457774 of database G:\ExchSvr\MDBDATA\priv1.edb. Error -1018.
– Event ID 454, Source ESE, Category: Logging/Recovery, Message: Information Store (1220) First Storage Group: Database recovery/restore failed with unexpected error -1018.

[And several more such logs]

And from within Exchange System Manager, the stores simply wouldn’t mount. There was no real error message.

The first thing I did was to make a copy of the MDBDATA directory. When doing any work to the Exchange database files, ALWAYS MAKE A COPY BEFORE YOU START!

Then I found some instructions on using eseutil to repair the databases – and I ran through them and ESEUTIL claimed it had fixed the problem. The instructions are found here: http://www.mike-tech.com/article.php?gif=exchange2k3&article=304

However, the mount of the database still wouldn’t work, with more of the same errors.

After more Googling and trying other things, I found this article: http://support.microsoft.com/kb/264228/en-us

Okay, so it points to missing database files, where the solution is to put them back where they belong. mmmmmm… Or the other solution is to run eseutil /r. Which I did – and it didn’t work either.

A similar article is also found here: http://technet.microsoft.com/en-us/library/bb498209(EXCHG.80).aspx

Then I took another copy of the MDBDATA directory, and decided to do a full restore from the last full backup – which was done the night before. So in the worst case scenario, we’d lose one day of emails – which for us, is fine. I could then look at the log files later.

So after doing a full restore, and following the brilliant instructions in our white paper which I’d written a few years prior (available here http://www.backupassist.com/downloads/whitepapers/ExchangeMailboxBackup_WP.pdf), I was all set! Or so I thought…

The blasted Information Store still wouldn’t mount!

Argh – how could this be??? A full backup wouldn’t mount?

After some thought, I deduced that the problem was that when the Information Store service starts, it automatically tries to use the log files to bring the database “up to date”. And it’s most likely that the log files were corrupt.

So I deleted the two *.log files [E00.log and E00xxxxxx.log] and did another full restore.

And hey presto! The mailbox store mounted successfully!

Now, did we lose any emails, given that we restored from the previous night’s backup? Well, in actual fact, no. We use Outlook in Cached Exchange mode so all the emails sent and received during the day were cached. And to my surprise, when we connected Outlook back to the Exchange Server and it synchronised, it did NOT delete the emails that were only in the OST file but not on Exchange. Having said that, those emails weren’t pushed back to the Exchange Server either. For us, this situation was acceptable.

So the total time taken to do this was about 1 hr 30 mins. Of that time, about 1 hr 15 mins was spent pursuing the eseutil database repair options… another 10 minutes figuring out that I had to delete the *.log files, and 5 minutes actually doing the restore.

The magic is knowing what to do… so I hope this information will be of benefit to other poor souls who suffer the same fate!

And remember – always backup! Thanks BackupAssist!