Last Tuesday morning was not good for me. As the title of this blog suggests, a lot went wrong, and it happened all by itself. It never ceases to amaze me how Microsoft systems can break all by themselves.
We have a SBS 2003 network, fairly small (about 15 workstations) with a very standard configuration of just Exchange, file sharing and RWW. You’d think that not much could go wrong…
Upon logging into my workstation last Tuesday morning, I was presented with a black screen and a 10 minute wait accompanied by a flurry of hard drive activity. I wasn’t the only person this was happening to but I put it down to an automatic update installing from the previous day. I began my day by checking Outlook and was very pleased to find that nothing had arrived since I had logged out the previous day. I sent a few internal emails, and then followed them up in person only to find that none had arrived. About now my other colleagues were starting to notice the same thing.
I strolled down to the server room to see what was causing the mail backlog. I opened the Exchange System Manager and after a lengthy pause I was greeted with “The Specified Domain Either Does Not Exist or Could Not Be Contacted”. From here, it quickly became apparent that the issues were much more widespread. During the course of the night Active Directory had become corrupt and was unable to identify our server as the DC, instead, looking for a backup DC that had been decommissioned more than 12 months earlier!
How can something like this happen? Everything was working just fine until last Tuesday.
We began by trying to rebuild the Active Directory database. We spent over two hours attempting this process, but to no avail; we were unable to point Active Directory back to the correct server. As all of our applications were functional and we had lost no data, we decided to take a different tack and perform a System State restore. Our DC is running Windows 2003 SBS, so we had been using BackupAssist to create a System State backup each night through the NTBackup Engine. We attached the USB HDD with a backup of the System State from the previous week and began the restore process.
Fortunately, the restore was a very straightforward process and we were running again in about 15 minutes.
The first step was to launch BackupAssist, navigate to the restore tab and select “NTBackup restore”. Then it was a simply matter of selecting the date of the backup we wanted to restore, choosing the System State item from that backup and beginning the restore process by completing the restore wizard. This process was fully automated and only required a quick reboot to update the files and registry information. Our server rebooted successfully, now with Active Directory up, running and correctly identifying our server as the Domain Controller. From here we were able to fully remove the items that were still pointing to our decommissioned server, ensuring this issue didn’t reoccur.
The important thing that came out of this nightmarish situation is how much faster and easier it is to solve these types of issues when you have a comprehensive backup plan. Without the System State backup that we had been performing nightly, we probably still would have been rebuilding our primary server!
Although this was performed in a Windows Server 2003 environment, however it could just as easily been done on a Windows Server 2008 environment through the BackupAssist Windows Imaging engine. It just so easy to tick that extra option to perform your system state backup, there is no reason why you shouldn’t.
The moral is: I, like many others, under value the importance of a stand-alone System State backup. The real value of this backup isn’t seen until you a faced with an issue within Active Directory, or the registry, that doesn’t extend to your applications or data. Without a System State backup that can be restored independently, you would be faced with having to perform a full restore of your server to a previous day; losing data and wasting time.
3 thoughts on “Exchange dead! Domain login unbelievably slow! Network shares disconnected! Domain Controller could not be located..? Thankfully we had a System State Backup…”
I would like to add that a system state restore will only work on the same hardware. The real issues occur when your server is dead and you have to replace it with a new one containing different hardware.
The problem with a “bare metal restore” on different hardware are the drivers. All the installed drivers on the old server are restored to the new server resulting in a crash when it tries to boot. Even a system state backup can’t resolve this.
Only backup software with the capability to install new harddisk/controller drivers during the restore process will resolve this issue.
With that kind of software, the only option is a clean install of the new server. With no 2nd DC in the network – active directory would be lost as well.
Last paragraph should read:
WithOUT that kind of software, the only option is a clean install of the new server. With no 2nd DC in the network – active directory would be lost as well.
shimrod… You’re right, but a basic SystemState backup even weekly can save you a ton of time in situations where AD or the registry got damaged.
Along those lines though, I wish BackupAssist utilized something like DriveImageXML at very least, to come up with a full-image-based backup for Windows 2003 installations. Of course, this doesn’t do bare metal recovery to DIFFERENT hardware; you’d have to use Backup Exec System Recovery or Acronis on WS2003. However, BackupAssist really has a ton of compelling features, and makes life a lot easier than just using the basic built-in Windows Backup.