Thursday, 28 June 2012

Simulating Exchange 2003 load with LoadGen

Next I wanted to simulate some load on the Exchange 2003 deployment.  To do this I wanted to use the Microsoft LoadGen application, and install it on a temporary, dedicated Virtual Machine.  I therefore deployed a new VM, and so that I wouldn't need to worry too much with the performance of the LoadGen machine itself, deployed it with twice the memory and CPU count as the Exchange 2003 back end.  As it's only temporary VM, I don't need to worry about this high allocation eating into available resources too much.
Once the server was built I RDP'd to it and downloaded the 32-bit version of LoadGen available here.
You also need to install the 32-bit Exchange 2007 management tools available here.
The management tools install insists on doing the 2007 schema updates, despite the fact it's just the tools I'll be installing.  I'll be installing Exchange 2007 in the not too distant future, so it's not a huge issue.
The installation of the Exchange tools completed successfully.
With the Exchange Tools installed I could begin the installation of LoadGen.
The installer is pretty straight forward, accept the licence agreement, choose the install location and it completes.
I could then launch LoadGen and opt to start a new test.
I'll be creating a new configuration.
I kept the test duration with the default times of a 10 hour run and an 8 hour day, and entered the Directory and Mailbox passwords.
To keep the numbers nice and simple, seeing as I have 1GB mailbox limits, and a 50GB database drive, and 3 databases I opted for 30 users total, with 10 per database, which would take (if fully utilised) the 50GB drive up to 30GB used, leaving 20GB for logs and public folders.  As long as I don't generate more than 25GB of logs total before I perform the first backup and log truncation this should be fine.
Next, I added some distribution lists and contacts for the simulation, but avoided any settings that may actually generate external email.
LoadGen then creates the test AD objects.  I then loaded up dsa.msc, and created 4 OU's in the LoadGen users OU, one for each type of Outlook 2003/2007 cached/non-cached combination, and moved the LoadGen created users via simple round robin into these four OU's.  This way I get a split between user profiles on the same database.
Then, back in LoadGen, I created 4 user groups, pointing at each of these OU's, and configured all of them as very heavy profile.
I'm only generating load from this system, so don't need to configure remote load generators.
Finally, you get a summary of selected options.
I saved the config so I could easily re-use it later, and started the initialisation phase.  A few hours later the initialisation phase completes, and I am left with a 20GB utilisation of the 50GB data drive (10 of the 20GB being logs).
I also ended up with a nice spread of users across the three mailbox databases, in the 250-320MB region but all with an item count of exactly 7809.
Now, back in LoadGen, I could start the simulation.
Before the 8 hours of the simulated day starts, I took the opportunity to log onto the Exchange Back-End, and start up perfmon adding some basic counters.
RPC Averaged Latency is really the one I'm interested in, it's a very good indicator of how clients will perceive performance.  Anything consistently under 20ms and client performance will be excellent, and the slower database disk won't actually be a cause of significant problems.  Under 50ms and it might be a little laggy but basically usable.  I added the disk counters so that if RPC Averaged Latency is high, I can either put it down to the disk design, or if it doesn't show high disk queues and utilisation at the same time as high RPC Averaged Latency, it would point me to look somewhere else for a bottleneck.

With the simulated day well underway, I took another look at the perfmon.
I wasn't even seeing Averaged RPC Latency spike above 10ms.  That's pretty excellent, and eradicated the minor concerns I had over disk performance due to the results I saw from JetStress earlier.

Many hours later, the tests completed and a summary was presented.
I was fairly pleased with the results, latencies all pointed towards this configuration being pretty solid for the kinds of load I'll be throwing at it.  The real question now is will it be able to keep those latencies under 20ms when I add a 2007 back end with another 50GB partition on the same disk.  That's still some way off yet though but the results should be interesting.

Tuesday, 26 June 2012

Testing the Exchange 2003 Disk Subsystem with JetStress

Now it's generally not recommended to run JetStress after Exchange has been installed, and in the case of this lab, JetStress results will be pretty much meaningless as I'll be adding more load to the same SAS disk that the Exchange database disk is on.  However, in the interests of completeness, and to get a general idea of how performance currently stands I decided to put the setup through its paces.
I don't want to be messing around with the thin provisioned disk the databases currently sit on, so I decided to add a additional disk with the same config as the current database disk, which I can delete once the test has been run.
I then RDP'd to the Back-End server and initialised and brought online the disk.
Next, I needed to download the 32-bit version of JetStress which can be obtained here.  The installer is pretty much an next, next finish scenario.  You also need to put a copy of the following four files in the JetStress directory, in this case as I already had Exchange installed, I could simply copy them from the Exchsrvr\bin directory, but you can also pull them off the Exchange install ISO.
The first time you run JetStress, as the advanced performance counters aren't enabled, JetStress enables them and needs to be restarted.
I don't have a defined test scenario, so I'll be creating a new one.
For now, I just want to test disk subsystem throughput.
I capped the storage capacity percentage at 80% and left the target IOPS at 100.
I'll be running a simple performance test here.
I kept the output file for results at the default location, and set the test to run for two hours.
I'll just be simulating a single Storage Group and Database, and I pointed the database at the drive I added earlier.  This version of Jetstress won't let you simulate logs and database on the same disk, despite the fact that this configuration can be set without problems in ESM.  The fact there was no log and database on the same disk warning in ExBPA leads me to believe it's a supported configuration (it is for 2007 and 2010) and as long as you have been sizing and speccing your disks correctly there shouldn't really be an issue. (Although getting enough performance from this kind of setup would have been more difficult with the available hardware when Exchange 2003 was first released)... Anyway, I created a tempory logs folder on the existing data drive.  This means the results will be somewhat removed from what I should expect with the actual live configuration, but should be somewhat similar to what I would get if I added an additional logs drive to the live system and moved the logs there.
The final configuration option allows you to select a database source or create the database, I opted to create it, then you are presented with a summary of the options selected.
After clicking execute test, the test begins.  With the options selected the test tries multiple thread counts to try and achieve a pass.  It didn't get one, but I wasn't really expecting it to, and in actuality, the latencies seen in the results aren't as bad as I thought they may be.  For the types of load I'll generally be putting on the system I'm not expecting these numbers to cause a huge issue.  The result inside the JetStress window is in a text box of limited size, even when maximised, so I pasted it into notepad for easier reading.
There's little point in me attempting to re-run the test with set thread counts - it's unlikely to make it pass, but as above, I'm not too worried about these numbers, the little client and OWA testing I've done so far has left me happy that the performance is adequate for the purposes of this lab, and the numbers above hint towards the possibility that I may be able to (perhaps with some minor design tweaks) get a pass for 2007 or 2010 (not that the lab needs to pass this validation - it's just a nice to have).

Saturday, 23 June 2012

Validating the Exchange 2003 build with ExBPA - Part 2

Next, lets look at the informational alerts that showed up in the ExBPA scan.
Now the Application log size is recommended to be increased to at least 40MB.  For a production platform this is a very good idea, particularly if you ever need to turn up diagnostic logging to troubleshoot an issue.  I'm not really expecting a huge throughput in the application log for this lab environment, although I will be performing a number of tasks that my well require me to turn up diagnostic logging levels, so it's worth doing in this case too.  Rather than individually increase the log size on Exchange servers, I've decided to increase the log size across all servers, via Group Policy.  This option will be particularly useful as the same recommendation applies to Exchange 2007 and 2010, so making the single change in Group Policy will mean not having to worry about this as I add more Exchange Servers.
I won't go into details about how I configured the GPO, as I've done this a couple of times already.
The precess of Tar Pitting is where unauthenticated SMTP connections are artificially slowed if the sending SMTP server addresses an invalid recipient is in an attempt to reduce the effectiveness of dictionary email address harvesting attacks as described in this Microsoft KB article.  It's not really relevant in this deployment as I won't be opening inbound SMTP to the Internet whilst only Exchange 2003 is deployed, however, it's an easy fix to reduce the number of informational alerts.  This is enabled by adding a DWORD value with number of seconds to delay at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SMTPSVC\Parameters.
The alert for the automatic updates not being enabled for the Intelligent Message Filter appears to be another bug.  The Intelligent Message Filter is not enabled on this server: the reg keys that ExBPA should be checking to verify if this is the case, as documented in this technet article don't even exist on the server.  I could probably fix the alert by enabling the updates for the service, but I'd be concerned that doing so would be more likely to introduce other issues so I'll have to live with this alert.
Basic Auth is enabled on the SMTP VS on the Front-End server by design.  It's not going to be open to the Internet so there is no security concern.  I should be able to disable Basic Auth on the back end SMTP VS though.
The circular logging alert for the Front-End relates to the single (empty) database that needs to exist on an Exchange 2003 Front-End server in order for it to properly function.  Circular logging is fine for this, so I modified the setting at the Storage Group Level in ESM.
Finally, crash upload logging is disabled.  In corporate environments, particularly in the EU, as it is pretty much impossible to know what information may be contained in a crash dump leaving this disabled is pretty much the only viable option.  Enabling it potentially raises concerns around transferring data outside of the EU, and the various data protection regulations that must be considered.  Crash dumps can still be created, and manually submitted to Microsoft after manual review if required.
Finally I re-ran the ExBPA scan to ensure everything I'd intended to fix was now fixed, and the only remaining items were expected.
The scan came back as expected. 13 items remained of the original 21, and the 13 will drop to 12 when I put some backup solution in.

Validating the Exchange 2003 build with ExBPA

For the next step in the lab build I wanted to look at performing a ExBPA scan, and correct any issues that show up.  It's often easy to pass off certain results of an ExBPA scan with X is expected because of Y or X isn't actually a problem due to Z. However, I find it's generally a good idea to clear down as many of the issues as possible, to minimise the time it takes to review future scans, and make it less likely that something that could be an issue getting overlooked should it show up in the future.  I first downloaded and installed the latest version of ExBPA designed for Exchange 2003 available here.

Lets first look at the critical results.
Now the backup critical error will be resolved when I implement a backup solution, and there's not a lot that can be done about the VMware detected error. But I should be able to correct the Missing FQDN in service principal name.
The setspn tool was installed as part of the support tools that were installed prior to Exchange, so it was just a case of navigating a command prompt window to the support tools folder, and running the setspn tool.
Next lets look at everything that showed up as a warning.
The network and storge drivers are the defaults than get installed with VMWare, I had a little bit of a hunt around but it doesn't appear to be anything newer around, however, given the size of the install base of VMWare, and even the drivers specifically being used here I would have expected any performance, bugfix or security updates to be readily available, so I'm not particularly concerned about these four warnings.

The SMTP configuration warning on RSMSGE11FE01 relates to the msExchSmtpMaxMessageSize being modified from it's default value of 0.  As per the technet article here, this can be ignored as this server is configured as a bridgehead for Internet mail.

The two IMAP4 fast message retrieval errors aren't an issue as I'm not planning on enabling IMAP for Exchange 2003 in this lab, however, enabling fast message retrieval on the (disabled) IMAP4 virtual servers will provide an easy win to get the warning count down.
This seems to be a Bug in this version of ExBPA, as apparently no warning should be shown unless all of the following are true;
  • The computer is running Exchange Server 2003.
  • The computer is not configured as an Exchange Server 2003 front-end server.
  • The IMAPv4 service and virtual server are running.
  • IMAP4 Fast Message Retrieval is not enabled on the Exchange Server.
Now in the case of the back end here, there is one criteria is the list that isn't met, and in the case of the Front-End, there's two. Anyway, Fast message retrieval on the Back-End can be easily enabled by going into properties of the IMAP4 virtual server in ESM, and ticking enable fast message retrieval.
The problem with the Front-End is that, being a Front-End, this option isn't relevant and ESM the tick box is greyed out.  There is however, a ADSIEDIT hack, you need to connect to the configuration partition and navigate through the tree to the IMAP4 virtual server instance.
Once there, you need to go into properties and find the returnExactMsgSize attribute and change this to False.
The SMTP performance warning on the Back-End is down to the Mail Queue being located on the system drive.  For this particular setup, I'm really not expecting mail queues, and even if there were, they would be much more likely to occur on the Front-End.  The level of concern over this really doesn't warrant adding an additional disk. The system disk is located on the SSD's so performance wise it would be better there than on the Database disk which is on the SAS storage.  The other option would be to put the queue on the 8GB PageFile disk, which, given a PageFile size of 2058MB has 6GB free. This seems like a reasonable compromise to get rid of the alert, given the low likelihood of significant mail queues forming on the Back-End compared to the Front-End is low, and the fact that I only gave the Front-End a 10GB Queue drive.
So I went into the properties for the SMTP VS on the Back-End and moved the Queue and Badmail directories to the PageFile disk.
Finally the WINS primary is Blank warning can be ignored as per this technet article, as the deployment is simple and on a single subnet, so NetBios name resolution shouldn't be a problem, and deploying a WINS environment just to fix this warning would be overkill.
I'll cover working through the informational alerts that came back in this scan in a subsequent post.

Thursday, 21 June 2012

Exchange 2003 - Setting up a redirect and making the SSL config secure.

The first tweak I wanted to carry out was to set up a redirect from the root of the default website, the best way of doing this (for a number of reasons) was covered quite nicely by ExchangeGeek, so I wont go into details aside from to say I modified the script to redirect to /exchange rather than /owa, as the linked article covers exchange 2007 and 2010 rather than 2003.

Next, as SSL 2.0 should no longer be considered secure, I wanted to disable it.  This can be done manually on a per server basis by modifying the registry settings at HKey_Local_Machine\System\CurrentControlSet\Control\SecurityProviders \SCHANNEL\Protocols, as described in KB187498, but a much better idea is to use Group Policy to ensure all IIS sites comply with the setting automatically.  SChannel settings aren't something that show up by default in group policy management, however, someone has created a custom adm file that can be imported and the settings managed from there.

So after loading the group policy management console, on one of the Domain Controllers, I created a new Group Policy named SChannel GPO.

After clicking on edit, you can then right-click the Administrative Templates node and choose Add/Remove templates.
After downloading the adm file, I added it in the screen above, it then appears under classic administrative templates, this is due to it being a adm file rather than the newer admx file typically used with Windows 2008.
I then disabled SSL 2.0 for both client and server requests.
And whilst in there I also took the opportunity to disable some of the less secure ciphers too.
I wanted a way to test that the change was successful, but actually perform connection tests, rather than just verifying the registry settings had been updated. Rather than download and install some utility such as nessus or metasploit to do this, I created a temporary public DNS record and firewall rule and ran the SSL tester at  There are quite a few sites out there that will perform SSL tests, but this seems to be one of the more comprehensive.
The first run (at least the part I was interested in) came back as follows;
I then ran a gpupdate /force on the exchange 2003 Front-End, followed by an IISRESET. I then re-ran the test and got back the following;
So it looks like the group policy is working as desired.