Monday, April 28, 2008

Nightmares with Exchange 2003 Free/Busy Time

I hadn't been in my current position long before I started noticing strange problems with our Public Folder structure. When I accepted my position as the Network Administrator I became the lucky winner of a battered 2003 Exchange server which was migrated over from version 5.5 back in ... 2005 or so.

I suppose the first sign that something was wrong was that there were strange little entries in the Public Folder list that didn't do anything. When you clicked on them in Outlook a message said "Unable to display the contents of this folder". I used to wonder what was in there and why I couldn't see it. But after asking around I could see that these were "artifacts". That's a nice way of saying, "junk that was left behind and for reasons beyond explanation - they will remain to exist for the life of the server". I tried to delete them. Operation failed. Crap. Now I didn't really care what was in them. That fun curiosity had left me. I just wanted them to go away.

And then came the Free/Busy issue. I had one of our more important and highly scheduled administrative folks ask me why people could schedule her to meetings when she was clearly booked solid. It only took a little bit of clicking around to see that something was wrong here. She was booked all day long but her "free time" only reflected an hour of busy time. I began with trying all the easy fixes, starting with "outlook.exe /cleanfreebusy" from the command line. It ran, without error. It fixed nothing.

Then I stared digging through an endless search of forums, newsgroups, and "knowledge-less bases". It seems I am not alone in my quest for operational Free/Busy functionality. There are many out there that like me are having the same types of issues. I saw a lot of people asking "can't I just delete this Free/Busy time and start it over"? The answer is, no. Because Free/Busy time is a system folder which lives in Public Folders, it isn't easily accessible by anyone. From within the Exchange System Manager you can navigate to it, play with it's simple permissions, and check it's replication. But that's about it. Oddly enough, I didn't really see anything wrong with this folder, despite the fact that it was totally broken.

I also publicly displayed my frustration in a newsgroup, which bared no helpful advice whatsoever.

After reading techno-babble off and on for the better part of two days I came across something of interest. It asked me to check an attribute on the Exchange server using the adsiedit.msc tool. Lo and behold I had located a problem. A particular attribute still held a link to my dead Exchange 5.5 server. Fixing this broken link would theoretically release, recreate, and reattach my Public Folder infrastructure. And so the following weekend I stayed up late drinking diet soda and hacking up my server only to find that the problem STILL EXISTED.

So here is the solution for anyone else that might end up in this mess. First I should mention that I have held back on releasing this entry until today (even though I performed this work nearly a month a ago). The details are a bit fuzzy to me now, but I didn't want to post a solution that wouldn't work. Today, I can honestly say that everyone's Free/Busy time is in good standing - and all those odd-ball Public Folders have been done away with.

Following this procedure will blow away your Public Folders completely, leaving nothing behind. The majority of this process covers how to backup and restore the data that your users will want back. This is a risky procedure, so if you try to do this and break things really badly - don't come looking for me. You have been warned.

Another important note: Performing this procedure will break "favorites". That means if your clients have opened up Outlook and said "add this folder to Favorites", they will now have a broken link. Even if your folder comes back with the same name and the same location - the shortcut will still not work. They (with your help) will need to recreate all of those shortcuts. Expect calls. Clicking on a dead shortcut will cause Outlook to crash!!

Step #1 - Make a backup. If you use Veritas, Symantec, or something of that nature - make a full backup of your Public Folders now. Hope you will never need it. In fact, try not to ever use it. Refer to the notes at the end of this post.

Step #2 - Back up the Public Folder permissions. Get a copy of the Microsoft Exchange Server Public Folder DAV-based Administration Tool. Install this tool (it extracts to a folder) and run it. Then click File > Connect. Enter the properties for your server, and run this as someone with Administrative access. Make sure that the radio button option is selected for "Public Folders". Now you should be able to expand Public Folders, and see them all listed in the left pane. Click on the very top item "Public Folders", and then click Tools > Export Permissions. Leave things at default, and click OK (you may have to set up a log file, so create one if prompted). This will create a text file with all of the Public Folders names, and all the permissions to go with them. In my case, I then opened up this text file ... went down to where it switched from "real" public folders to invisible System Folders. Then I *deleted every line* which referred to System Folders. You should do this too. Problems with Free/Busy could be related to incorrect permissions being applied to your folder set. You do not want to re-import those faulty permissions back onto a healthy Public Folder store.

Step #3 - Back up your Public Folder data. I did this the old fashioned way. By that, I mean that I opened up Outlook, expanded Public Folders, then selected All Public Folders. Then I performed a File > Export, and exported *everything* in Public Folders to a PST file. The danger in doing this is: you cannot back up folders that you have no permission to. So if someone has excluded you access to a folder, it will not get backed up. That could get you in trouble. Compare what you see in Step #2, with what you see inside of Outlook. Make sure you are not missing anything. Also know, this could take *HOURS* depending on the amount of Public Folder data that you have. In my case the store was a little less than 700MB and it took 45 minutes.

Step #4 - Remove Public Folders. To do this, go into Services on the Exchange Server and stop the Information Store. Now, browse to where the Exchange data files are physically stored. Usually this is in x:\Programs Files\Exchsrvr\mdbdata. There are two files; pub1.edb, and pub1.stm. Rename these files - but do NOT delete them. I just added an .old extension to them. Now, go back and restart the Information Store service. This will cause chaos and confusion to your Exchange Server. It should give you a bad news message and ask if you want to create a blank Public Folder set. Say yes. Congratulations, you just destroyed all of your users data. Better act quickly on step #5.

Step #5 - Put the Public Folder data back. This is the reverse of exporting. Go into Outlook, Expand Public folders and notice that it's empty. Now, import your PST back to Public Folders. Note that there is a trick to this! The trick is, you cannot import back into a system folder without Outlook telling you to "stop doing that". What you can do, is expand Public Folders, then expand All Public Folders - and then start the File > Import wizard. At the second or third step where you tell it where you are importing to - select "to currently selected folder". You will also notice that in your PST file this subtree has some bizarre name like "IPM_NON_SUBTREE". Don't worry about that. It will restore to where it needs to. Watch the files copy. When done, make sure things look okay.

Step #6 - Still awake? Now fix the permissions. Open up your tool from Step #2, and click File > Connect. Again, fill in your server properties and make sure Public Folders are selected. Now select Public Folders and then Tools > Import Permissions. This should go pretty quickly. To see if things worked, you should go right clicking randomly on your folders and make sure that the permissions look right. You can also check the log for this tool you have been using.

Step #7 - Update everyones Free/Busy time. The best way to do this is to send out a mass meeting notice for a "fake meeting". You can put it for a Sunday at Midnight, make it last 10 minutes, and put the location down as "fairy land". What's important is that everyone in your organization gets it, and agrees to the meeting. Doing so, will reset their free/busy data on the server. You can also recreate this data by having each user run Outlook from a command line with the /cleanfreebusy switch. Good luck with that! I used the fake meeting method, and it worked wonders.

That should be all there is to it. But there are some ...

Possible Pitfalls!
Free/Busy still not accurate - Let's say that you have imported everything, fixed all the permissions, and the Free/Busy is still whacked. Take a moment to think about this. We have fixed all future appointments, but existing ones may still be a problem. I found that if you delete a reoccurring appointment and recreated it, the free/busy became accurate. Also - waiting longer seems to work. I waited about two weeks and everything seems correct. What fixed it? Who knows. It's Exchange Server.

Folders are missing - If someone had a folder which you could not access, than you probably didn't back it up. Way to go! The good news is that you *renamed the data files*, you did not delete them. The bad news is that you will have a hard time getting the data out of them. If at all possible - do NOT restore from a tape backup. The best thing you can do is use a tool to extract the data from the public folder files. One such tool is called OnTrack PowerControls. It's expensive to buy, but you should be able to use the trial version to extract from a detached Information Store database file. Basically you need to extract the data into a PST, and then import that PST back to the Public Folder tree. If you are stuck doing this ... read the manual for the PowerControls product. ;-)

I sincerely hope that this information comes in useful to someone, some day. It took me a few weeks of off and on experimentation to come up with this. If this helped (or harmed) you won't you please drop me a comment and let me know?

-Steve Ballantyne