So I just installed RU1 on my brand new Exchange 2010 server and then I issue a Get-Exchangeserver -Identity MyExchangeServer and get the following output for AdminsDisplayVersion and ExchangeVersion:

Ok that looks a little familiar for some reason. I go to my Exchange 2010 RTM server and issue the same CMDlet and get:

...The same result! But one server has RU1 installed and the other is RTM. Shouldn't I get a different version number back?

Well... no. Exchange 2007 and forward do not reflect the version number either in the value for AdminDisplayVersion, ExchangeVersion, or at this registry key HKLM\SOFTWARE\Microsoft\v8.0\<Role>\ConfiguredVersion as influenced by roll ups. This is a common misconception.

The most conclusive way to get the version of your exchange server, rollup and all, is to check the file version of ExSetup.exe in the BIN folder.

Here is Exchange 2010 RU1 version:

And here is Exchange 2010 RTM:

Another way of getting this information is to run the following PowerShell one-liner:

GCM exsetup |%{$_.Fileversioninfo}

The below output is from an exchange 2010 server running RU1:

Here is an exchange 2010 RTM server:

You can then correlate the version number you find with those listed here, here or on the actual rollup update download pages.

Hope this post reduces some confusion out there!

- Tom Kern

Today I am pleased to announce the release of the Exchange Server Pre-Deployment Analyzer (ExPDA).

You can use the Exchange Pre-Deployment Analyzer to perform an overall topology readiness scan of your environment. When you run the Exchange Pre-Deployment Analyzer, it provides a detailed report that will alert you if there are any issues within your organization, which could prevent you from deploying Exchange 2010. For example, the Exchange Pre-Deployment Analyzer will notify you if you haven't deployed the minimum required Exchange service pack on all your existing Exchange servers.

The checks performed by ExPDA are similar to the pre-requisite checks implemented (via Exchange Best Practices Analyzer) in the Exchange 2010 Setup program; in fact ExPDA is based off the Exchange Best Practices Analyzer (ExBPA) engine. However, unlike Exchange 2010 setup, this tool focuses only on overall topology readiness and not the ability to run Exchange 2010 on the local computer. The scan also performs a deep analysis of each existing Exchange 2003/2007 server to verify that it has the necessary updates and configuration in-place to support Exchange 2010. The end report is structured as follows:

• Critical - A configuration problem that will prevent Exchange 2010 from being deployed in the organization. For example, the Active Directory Forest is not operating in Windows Server 2003 Forest Functional Mode or higher.
• Warning - A configuration item that may prevent customers having the best possible experience with Exchange 2010. A warning may also reflect some functionality that is not available in Exchange 2010.

ExPDA is another component in our vision to provide a seamless upgrade experience that reduces the complexities in deploying Exchange 2010. To start planning your upgrade, please utilize the Exchange Deployment Assistant. The Deployment Assistant allows a customer to create Exchange 2010 on-premises deployment instructions that are customized to their environment. The Assistant asks a small set of questions, and based on the answers, it provides a finite set of instructions that are designed to get a customer up and running on Exchange 2010.  Running the Exchange Server Pre-Deployment Analyzer is now a recommended step within the pre-requisites section of the Deployment Assistant.

You can download ExPDA at http://www.microsoft.com/downloads/details.aspx?FamilyID=88b304e7-9912-4cb0-8ead-7479dab1abf2&displaylang=en.  ExPDA is supported on Windows 7, Windows Vista with Service Pack 2, Windows Server 2008 with Service Pack 2, Windows Server 2008 R2, and Windows Server 2003 with Service Pack 2.

Q&A

1. Does this tool replace the Exchange Deployment Assistant?

   No, ExPDA is merely an additional tool that can be used as a step within the upgrade experience. The Exchange Deployment Assistant will walk you through all aspects of the upgrade, namely how to coexist properly with Exchange 2010 and legacy versions of Exchange, whereas, ExPDA is one step within that process and ensures that the environment is ready to have the first Exchange 2010 server deployed.

2. I ran the scan and have questions about the results. What should I do?

   If you'd like to read more about the requirements of Exchange 2010, please see the Planning for Exchange 2010 section on TechNet.

3. I ran the scan and received unexpected results. If I think there's a bug, who can I contact?

   If you need assistance, please visit the Exchange Server Deployment Forum or you can send mail to exbpafb AT Microsoft DOT com.

4. Is this new functionality available in all languages?

   No. ExPDA is only available as a U.S. English version.

5. I'd like to know if my organization is capable of running Exchange 2007. Can I use ExPDA to check this?

   No, ExPDA only verifies if an organization is ready to have the first Exchange 2010 server installed. If you need to determine whether your organization is ready to have Exchange 2007 deployed, you can utilize ExBPA v2.8 and the latest Exchange 2007 ExBPA.Readiness.xml:

   1. Install ExBPA 2.8 and the last ExBPA 2.8 Update.
   2. Download the latest Exchange 2007 service pack rollup. At the time of this writing this is SP2 RU2.
   3. Extract the rollup binaries using this command: msiexec /a filepath to MSI file /qb TARGETDIR=filepath to target folder
   4. Copy the ExBPA.Readiness.xml to the \en folder.
   5. Launch ExBPA.

6. Where can I look at the list of checks made in this new scan?

   If you're familiar with the BPA XML files, the new check is wholly contained within the ExBPA.Readiness.xml file located within %Program Files%\Microsoft\Exchange Server\V14\ExPDA\en folder.

   • Reporting the number of Active Directory trees, domains, sites, admin groups, routing groups, Exchange 5.5 servers, Exchange 2000 servers, Exchange 2003 servers, total mailboxes, Windows 2000 Active Directory servers, Windows Server Active Directory servers, Windows Server Active Directory servers. Report how many Active Directory domain/sites have Exchange servers installed.
   • Verifying that the Schema Master is Windows 2003 SP1 or later.
   • Identifying Active Directory domains that are not in native mode.
   • Identifying Active Directory sites that do not have a global catalog server running Windows 2003 SP1 or later.
   • Verifying that there are zero Active Directory Connector servers in existence.
   • Identifying any SMTP site links in existence.
   • Verifying that the Exchange organization is in native mode.
   • Identifying any non-standard proxy address generators.
   • Identifying whether you have any ambiguously defined email addresses in your recipient policies.
   • Identifying any non-MAPI public folder hierarchies (a.k.a. AppTLH's) in use.
   • Identifying Routing Groups that span Active Directory sites.
   • Identifying any Active Directory sites that span Routing Groups.
   • Identifying any Routing Group Connectors that have specialized settings (activation, max size, accept/reject lists, restrict message type/priority).
   • Identifying any SMTP Connectors that support non-SMTP address spaces.
   • Identifying any SMTP Connectors that use inline domain-wildcarding for address spaces (e.g. *foo.com instead of *.foo.com).
   • Identifying any X.400 Connectors in the topology.
   • Identifying any EDK-based Connectors in the topology (excluding Notes).
   • Verifying that any servers running Exchange 2003 have SP2 or later.
   • Verifying that any servers running Exchange 2007 have SP2 or later.
   • Identifying any SMTP virtual servers that are not using port 25 for incoming/outgoing.
   • Verifying that all Exchange 2003 servers have SuppressStateChanges set.
   • Identifying any Exchange 2003 servers that have active NNTP newsfeeds.
   • Identifying any Exchange 2003 servers that use the Event Scripting service.
   • Identifying any Exchange 2003 servers that have the ExIFS (a.k.a. M:) drive enabled.
   • Identifying any parts of Active Directory that have Access Control Entry inheritance disabled.

Thanks for your continued support. Feel free to post feedback in comments of this blog post, or visit the Exchange Server Deployment Forum.

- Ross Smith IV

In Exchange Server 2010, there is no more single instance storage (SIS). To help understand why SIS is gone, let's review a brief history of Exchange.

During the development of Exchange 4.0, we had two primary goals in mind, and SIS was borne out of these goals:

1. Ensure that messages were delivered as fast and as efficient as possible.
2. Reduce the amount of disk space required to store messages, as disk capacity was premium.

Exchange 4.0 (and, to a certain extent, Exchange 5.0 and Exchange 5.5) was really designed as a departmental solution. Back then, users were typically placed on an Exchange server based on their organization structure (often, the entire company was on the same server).  Since there was only one mailbox database, we maximized our use of SIS for both message delivery (only store the body and attachments once) and space efficiency. The only time we created another copy within the store was when the user modified their individual instance.

For almost 19 years, the internal Exchange database table structure has remained relatively the same:

Then came Exchange 2000.  In Exchange 2000, we evolved considerably - we moved to SMTP for server-to-server connectivity, we added storage groups, and we increased the maximum number of databases per server.  The result was a shift away from a departmental usage of Exchange to enterprise usage of Exchange.  Moreover, the move to 20 databases reduced SIS effects on space efficiency, as the likelihood that multiple recipients were on the same database decreased.  Similarly, message delivery was improved by our optimizations in transport, so transport no longer benefited as much from SIS either.

With Exchange 2003, consolidation of servers took off in earnest due to features like Cached Exchange Mode.  Again the move away from departmental usage continued.  Many customers moved away from distributing mailboxes based on their organization structure to randomization of the user population across all databases in the organization.  Once again, the space efficiency effects of SIS were further reduced.

In Exchange 2007, we increased the number of databases you could deploy, which again reduced the space efficiency of SIS. We further optimized transport delivery and completely removed the need for SIS from a transport perspective.  Finally, we made changes to the information store that removed the ability to single instance message bodies (but allowed single instancing of attachments). The result was that SIS no longer provided any real space savings - typically only about 0-20%.

One of our main goals for Exchange 2010 was to provide very large mailboxes at a low cost. Disk capacity is no longer a premium; disk space is very inexpensive and IT shops can take advantage of larger, cheaper disks to reduce their overall cost. In order to leverage those larger capacity disks, you also need to increase mailbox sizes (and remove PSTs and leverage the personal archive and records management capabilities) so that you can ensure that you are designing your storage to be both IO efficient and capacity efficient.

During the development of Exchange 2010, we realized that having a table structure optimized for SIS was holding us back from making the storage innovations that were necessary to achieve our goals. In order to improve the store and ESE, to change our IO profile (from many, small, random IOs to larger, fewer, more sequential IOs), and to resolve our inefficiencies around item count, we had to change the store schema. Specifically, we moved away from a per-database table structure to a per-mailbox table structure:

This architecture, along with other changes to the ESE and store engines (lazy view updates, space hints, page size increase, b+ tree defrag, etc.), netted us not only a 70% reduction in IO over Exchange 2007, but also substantially increased our ability to store more items in critical path folders.

As a result of the new architecture and the other changes to the store and ESE, we had to deal with an unintended side effect.  While these changes greatly improved our IO efficiency, they made our space efficiency worse.  In fact, on average they increased the size of the Exchange database by about 20% over Exchange 2007. To overcome this bloating effect, we implemented a targeted compression mechanism (using either 7-bit or XPRESS, which is the Microsoft implementation of the LZ77 algorithm) that specifically compresses message headers and bodies that are either text or HTML-based (attachments are not compressed as typically they exist in their most compressed state already).  The result of this work is that we see database sizes on par with Exchange 2007.

The below graph shows a comparison of database sizes for Exchange 2007 and Exchange 2010 with different types of message data:

As you can see, Exchange 2007 databases that contained 100% Rich Text Format (RTF) content was our baseline goal when implementing database compression in Exchange 2010. What we found is that with a mix of messaging data (77% HTML, 15% RTF, 8% Text, with an average message size of 50KB) that our compression algorithms are on par with Exchange 2007 database sizes. In other words, we mitigated most of the bloat caused by the lack of SIS.

Is compression the answer to replacing single instancing all together? The answer to that question is that it really does depend. There are certain scenarios where SIS may be viable:

• Environments that only send Rich-Text Format messages. The compression algorithms in Exchange 2010 do not compress RTF message blobs because they already exist in their most compressible form.
• Sending large attachments to many users. For example, sending a large (30 MB+) attachment to 20 users.  Even if there were only 5 recipients out of the 20 on the same database, in Exchange 2003 that meant the 30MB attachment was stored once instead of 5 times on that database. In Exchange 2010, that attachment is stored 5 times (150 MB for that database) and isn't compressed. But depending on your storage architecture, the capacity to handle this should be there. Also, your retention requirements will help here, by forcing the removal of the data after a certain period of time.
• Business or organizational archives that are used to maintain immutable copies of messaging data benefit from single instancing because the system only has to keep one copy of the data, which is useful when you need to maintain that data indefinitely for compliance purposes.

If you go back through our guidance over the past 10 years, you will never find a single reference to using SIS around capacity planning.  We might mention it has an impact in terms of the database size, but that's it.  All of our guidance has always dictated designing the storage without SIS in mind.  And for those that are thinking about thin provisioning, SIS isn't a reason to do thin provisioning, nor is SIS a means to calculate your space requirements.  Thin provisioning requires an operational maturity that can react quickly to changes in the messaging environment, as well as, a deep understanding of the how the user population behaves and grows over time to sufficiently allocate the right amount of storage upfront.

In summary, Exchange 2010 changes the messaging landscape.  The architectural changes we have implemented enable the commoditization of email - providing very large mailboxes at a low cost.  Disk capacity is no longer a premium.   Disk space is cheap and IT shops can take advantage of larger, cheaper disks to reduce their overall cost.  With Exchange 2010 you can deploy a highly available system with a degree of storage efficiency without SIS at a fraction of the cost that was required with previous versions of Exchange.

So, there you have it. SIS is gone.

- Ross Smith IV

Please go to our Mailbox Server Role Storage Requirements Calculator updates tracking page to see what is in this new version!

A blog post explaining the calculator (updated for this new version) is here.

Comments welcome!

- Ross Smith IV

The Exchange 2010 install of the CAS role includes the installation of the legacy Exchange Virtual Directories. The Exchange, Public and Exchweb Virtual Directories are placed and configured directly in the IIS metabase, therefore these Virtual Directories are only local to the Server and no longer objects located in Active Directory.

Additionally, the Exchange 2010 New-OWAVirtualDirectory cmdlet no longer recognizes the -Name parameter and is hard coded to use "OWA" for the name. This behavior is by design.

NOTE: This behavior is different than that in Exchange 2007 where it was necessary to use the New-OWAVirtualDirectory in order to recreate the Exchange, Exchweb, and Public Virtual Directories.

NOTE: Running the cmdlet Remove-OWAVirtualDirectory will remove the Exchange, Exchweb, and Public Virtual Directories in addition to the OWA Virtual Directory. The New-OWAVirtualDirectory cmdlet will create a new OWA Virtual Directory as well as the Exchange, Exchweb, and Public Virtual Directories.

So what happens if we need to manage or recreate only the legacy Exchange Virtual Directories on an Exchange 2010 Server?

Since the legacy Exchange Virtual Directories are no longer listed and manageable in Active Directory, we need to use the appcmd utility (as we do to manage other Virtual Directories in IIS 7 and above) or add the Virtual Directories in IIS manually and modify the applicationHost.config file directly.

Option 1:

In order to recreate the legacy Exchange Virtual Directories using appcmd, take the following actions for each Virtual Directory (Exchange, Exchweb, and Public):

1) Open a command prompt, change directory to %SystemRoot%\System32\inetsrv

2) Replacing Default Web Site with the appropriate Site Name (if different) and <VDir> to the correct legacy Exchange Virtual Directory alias, Run the following commands:

• appcmd delete vdir /vdir.name:"Default Web Site/<VDir>" /commit:APPHOST
• appcmd add vdir /app.name:"Default Web Site"/ /path:/<VDir> /physicalPath:"<Exchange Install Directory>\V14\ClientAccess\Owa" /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:directoryBrowse /enabled:false /showFlags:"Date, Size, Extension, LongDate" /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:handlers /accessPolicy:Read /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:windowsAuthentication /enabled:false /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:anonymousAuthentication /enabled:true /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:digestAuthentication /enabled:false /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:basicAuthentication /enabled:false /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:defaultDocument /enabled:true /commit:APPHOST
• appcmd set config "Default Web Site/<VDir>" /section:httpRedirect /enabled:true /destination:/owa /exactDestination:false /childOnly:false /httpResponseStatus:Found /commit:APPHOST

NOTE: I recommend putting the above into a batch file and make 3 copies. Modify each batch to replace the <VDir> with Exchange, Exchweb, or Public.

Option 2:

In order to recreate the legacy Exchange Virtual Directories manually, take the following actions for each Virtual Directory (Exchange, Exchweb, and Public):

1) Open IIS Manager, expand Default Web Site, right click on the <VDir> and choose Remove

2) Highlight the Default Web Site, and choose View Virtual Directories in the Actions pane.

3) In the Actions Pane, choose Add Virtual Directory... Add the correct Alias (Exchange, Exchweb, or Public) and Physical Path which is the "<Exchange Install Directory>\V14\ClientAcess\Owa" and click OK.

NOTE: Default Web Site is set to Anonymous by default, therefore the newly created Virtual Directories should also be set to Anonymous. If not, then modifying the applicationHost.config below will correct the authentication.

4) Once the above is complete, open the applicationHost.config file using notepad which is located in %SystemRoot%\System32\Inetsrv\config\

5) Search for the following: "Default Web Site/owa/14". Once found, look for the </location>, After the </location>, copy and paste the following text renaming <VDir> to the legacy Exchange Virtual Directory alias:

<location path="Default Web Site/<VDir>">
      <system.webServer>
           <directoryBrowse enabled="false" showFlags="Date, Size, Extension, LongDate" />
           <handlers accessPolicy="Read" />
           <security>
                <authentication>
                    <windowsAuthentication enabled="false" />
                    <anonymousAuthentication enabled="true" />
                    <digestAuthentication enabled="false" />
                    <basicAuthentication enabled="false" />
                </authentication>
           </security>
           <defaultDocument enabled="true" />
           <httpRedirect enabled="true" destination="/owa" exactDestination="false" childOnly="false" httpResponseStatus="Found">
                <clear />
           </httpRedirect>
       </system.webServer>
   </location>

After recreating the legacy Exchange Virtual Directories and modifying the applicationHost.config, open a command prompt and run:

Iisreset /noforce

- Steve Swift

In this blog post, we will be highlighting some of the most common errors that may be seen when attempting to open the Exchange Management tools (Exchange Management Console and Exchange Management Shell).

To start off, you first need to be aware that in Exchange 2010, all management is done via Remote PowerShell, even when opening the Management Tools on an Exchange server. Where this differs from Exchange 2007 is that there is now a much larger dependency on IIS, as Remote PowerShell requests are sent via the HTTP protocol and use IIS as the mechanism for connections. IIS works with the WinRM (Windows Remote Management) service, and the WSMan (Web Services for Management) protocol to initiate the connection.

When you click on the Exchange Management Shell shortcut, a Remote PowerShell session is opened. Instead of simply loading the Exchange snap-in (as we did with Exchange 2007), PowerShell connects using IIS to the closest Exchange 2010 server via WinRM. WinRM then performs authentication checks, creates the remote session and presents to you the cmdlets that you have access to via RBAC (Role Based Access Control).

Since all Remote PowerShell connections go through IIS, we have identified some of the most common errors that may be exhibited when attempting to open the Exchange Management tools along with the most common causes of those errors and how to address these issues. We have attempted to list these in order of frequency.

Issue:

Connecting to remote server failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.

Possible causes:

1. Remote PowerShell uses Kerberos to authenticate the user connecting. IIS implements this Kerberos authentication method via a native module. In IIS Manager, if you go to the PowerShell Virtual Directory and then look at the Modules, you should see Kerbauth listed as a Native Module, with the dll location pointing to C:\Program Files\Microsoft\Exchange Server\v14\Bin\kerbauth.dll. If the Kerbauth module shows up as a Managed module instead of Native, or if the Kerbauth module has been loaded on the Default Web Site level (instead of, or in addition to, the PowerShell virtual directory), you can experience this issue. To correct this, make sure that the Kerbauth module is not enabled on the Default Web Site, but is only enabled on the PowerShell virtual directory. The entry type of "Local" indicates that the Kerbauth module was enabled directly on this level, and not inherited from a parent.

2. If the WSMan module entry is missing from the global modules section of the C:\Windows\System32\Inetsrv\config\ApplicationHost.config file, as follows:

<globalModules>
           <add name="WSMan" image="C:\Windows\system32\wsmsvc.dll" />

This will result in the WSMan module displaying as a Managed module on the PowerShell virtual directory.

To correct this, make sure that the WSMan module has been registered (but not enabled) at the Server level, and has been enabled on the PowerShell virtual directory.

3. If the user that is attempting to connect is not Remote PowerShell enabled. To check if a user is enabled for Remote PowerShell, you need to open the Exchange Management Shell with an account that has been enabled, and run the following query.

(Get-User <username>).RemotePowershellEnabled

This will return a True or False. If the output shows False, the user is not enabled for Remote PowerShell. To enable the user, run the following command.

Set-User <username> -RemotePowerShellEnabled $True

Issue:

Connecting to the remote server failed with the following error message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more information, see the about_Remote_Troubleshooting Help topic.

Possible Causes:

1. The http binding has been removed from the Default Web Site. A common scenario for this is if you are running multiple web sites, and attempting to set up a redirect to https://mail.company.com/owa by requiring SSL on the Default Web Site, and creating another web site to do the redirect back to the SSL-enabled website.

Remote PowerShell requires port 80 to be available on the Default Web Site. If you want to set up an automatic redirect to /owa and redirect http requests to https, you should follow the instructions located at http://technet.microsoft.com/en-us/library/aa998359(EXCHG.80).aspx and follow the directions under the section "For a Configuration in Which SSL is Required on the Default Web Site or on the OWA Virtual Directory in IIS 7.0".

2. The http binding on the Default Web Site has been modified, and the Hostname field configured. To correct this issue, you need to clear out the Hostname field under the port 80 bindings on the Default Web Site.

Issue:

Connecting to remote server failed with the following error message: The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic. It was running the command 'Discover-ExchangeServer -UseWIA $true -SuppressError $true'.

In addition, you may see the following warning event in the System log:

Source: Microsoft-Windows-WinRM
EventID: 10113
Level: Warning
Description: Request processing failed because the WinRM service cannot load data or event source: DLL="%ExchangeInstallPath%Bin\Microsoft.Exchange.AuthorizationPlugin.dll"

Possible Causes

1. The ExchangeInstallPath variable may be missing. To check this, go to the System Properties, Environment variables, and look under the System variables. You should see a variable of ExchangeInstallPath with a value pointing to C:\Program Files\Microsoft\Exchange Server\V14\.

2. The Path of the Powershell virtual directory has been modified. The PowerShell virtual directory must point to the \Program Files\Microsoft\Exchange Server\v14\ClientAccess\PowerShell directory or you will encounter problems.

Issue:

Connecting to remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests on the correct port and HTTP URL. For more information, see the about_Remote_Troubleshooting Help topic.

Possible Causes:

1. Make sure the MSExchangePowerShellAppPool is running. If it is, try recycling the Application Pool and check for errors or warnings in the Event logs.

2. Make sure that the user that is trying to connect is Remote PowerShell Enabled (see the first error for details on how to check this).

3. Make sure WinRM is properly configured on the server.

a. Run WinRM Quick Config on the server and ensure that both tests pass and no actions are required. If any actions are required, answer Yes to the prompt to allow the WinRM configuration changes to be made.

b. Run WinRM enumerate winrm/config/listener and ensure that a listener is present for the HTTP protocol on port 5985 listening on all addresses.

Issue:

Connecting to remote server failed with the following error message: The WinRM client received an HTTP status code of 403 from the remote WS-Management service.

Possible Causes:

1. The "Require SSL" option has been enabled on the PowerShell Virtual Directory. To resolve this, remove the "Require SSL" option from this Virtual Directory. The Exchange Management Tools connect over port 80, not 443, so if Require SSL is set, when a connection is attempted on port 80, IIS will return a 403 error indicating SSL is required.

- Ben Winzenz, Solange Trombini

In many organizations the skill sets that administer Exchange, Active Directory (AD), Domain Naming System (DNS), and the network infrastructure are often segregated. Since Exchange is heavily dependent on communication with AD and that communication must traverse the network, the ability for the Exchange Administrator to have a basic understanding of how each of these components can be troubleshot is useful. Furthermore, often times the DNS, network, and AD administrators are not familiar with how Exchange is dependent on their services and thus are not fully prepared to help an Exchange administrator isolate production issues. As such, by the end of this it is hoped that there will be enough information to allow the Exchange, DNS, network, and AD administrators to each collect the data necessary to identify the majority of the issues.

There is some required reading that is necessary in order to understand much of the content that follows. Rather than reiterate all of the pre-requisite knowledge here, reviewing the following content might be helpful:

• Ruling Out Active Directory-Bound Problems
• Monitoring Without System Center Operations Manager: Exchange 2007 Help
  • Most specifically - Monitoring Common Counters: Exchange 2007 Help
• Why you need Active Directory for Exchange Server 2007
• Measuring AD Performance
• Understanding and Troubleshooting Directory Access
• Exchange Server 2003 and Active Directory
• Mike Lagase : New ADAccess Performance counters included with Exchange 2007 SP2
• Performance Tuning Guidelines for Windows Server 2003
• Performance Tuning Guidelines for Windows Server 2008
• Microsoft® Windows ServerT 2003 Resource Kit

To start with, an understanding of what the performance counters "LDAP Read Time" and "LDAP Search Time" track is very useful (as identified in Ruling Out Active Directory-Bound Problems and Monitoring Common Counters: Exchange 2007 Help). One knows that once these counters approach or exceed the recommended thresholds, troubleshooting outside of Exchange needs to begin. Therefore, the question, "What are the potential contributors to high values for these counters?" is thus raised. Since these counters are installed with Exchange and are tracked within the DSAccess /ADAccess components, depending on the version of Exchange, there are a number of dependencies that Exchange does not have visibility into that the troubleshooter needs to be cognizant of when evaluating where the source of the delay is incurred. In order to provide a clearer picture of what needs to be considered, below is a walkthrough of a somewhat simplified list of events that happens after the clock starts ticking (reference How Active Directory Searches Work for greater detail):

1. WLDAP32.DLL receives the request from one of the Exchange processes. It has to locate a Global Catalog (GC) first.

2. DNS query traverses the network. Unresponsiveness or degraded responsiveness from DNS servers will degrade the overall performance of the query.

3. WLDAP32.DLL submits the query to the GC.

4. If not already started, a Transmission Control Protocol (TCP) session is established, the Lightweight Directory Access Protocol (LDAP) query traverses the wire.
Note: since TCP requires a 3 stage handshake before the session and in turn windowing can work, multiply network latencies by 3 to figure out how much time it takes just to establish the session (thus that 10 ms latency becomes 30 ms delay even before the Exchange server submits the query).

5. The TCP data transmitted up the networking stack to LSASS.exe listening on the LDAP ports.

6. The query is processed by the GC and searches through the database to return the results.

7. The data is sent out the Network Interface Card (NIC) on the GC.

8. The query is received via WLDAP32.DLL from the GC.

a. If there are multiple pages all pages need to be returned and stored in a data structure within DSAccess / ADAccess. Each of these requires traversal of the network and resubmission of the query.

b. If there are a large quantity of values in the requested attribute (think group memberships), these all need to be retrieved and returned. This results in multiple queries to the GC. This may require steps 1 - 5 to recur.

As can be seen from the above set of steps a lot of the performance delays are heavily dependent on network latency and network performance. In fact the only portion of that number that is actually impacted by the GC performance itself is step 5. As such, in addition to troubleshooting any potential performance issues on the GC, it is highly advantageous to collect network trace data when issues do occur, preferably from both sides of the conversation.

Summary of Data needed to troubleshoot

Exchange Server - Having this data from both sides is important for comparison purposes (i.e. is the processor spike on the DC correlating with the degraded performance in Exchange)

• Performance Counters specified in the referenced articles:
  • Exchange 2007/2010 - Monitoring Common Counters: Exchange 2007 Help
  • Exchange 2000/Exchange 2003 - Ruling Out Active Directory-Bound Problems
    Note: For Exchange 2003 add in MSExchangeDSAccess Domain Controllers\LDAP Search Time(All Instances) and MSExchangeDSAccess Domain Controllers\LDAP Read Time\(All Instances) so that the DC to be troubleshot can be identified.
• Network Trace from the Exchange server side

Note: To ease collection of the Exchange performance data reference "Mike Lagase : Perfwiz replacement for Exchange 2007"

AD Server -

• Performance Data -
  • Windows 2008/2008 R2 - Monitoring Common Counters: Exchange 2007 Help recommends starting the built in Data Collector Set "Reliability and Performance\Reports\System\Active Directory Diagnostics" and use the data collector set.
    Reference: Performance Logs and Alerts
  • Windows 2003
    • At a minimum, the counters specified in Ruling Out Active Directory-Bound Problems
    • SPA - Select the Active Directory report.
• Network Trace from the affected DC side.

DNS Server - not all organizations use Microsoft DNS servers, but for the purposes of this article the assumption that a Microsoft DNS server is in use.

• Network Trace from the DNS server
• Performance Data
  • Windows 2008/2008 R2 - start the built in Data Collector Set "Reliability and Performance\Reports\System\System Performance" and use the data collector set.
  • Windows 2003 - SPA Select the System Overview report
  • Performance counters - All DNS related counters.

How to triage where troubleshooting needs to happen

• Network Infrastructure issues -
  • Use the network trace to determine if the DNS server is responding in a timely fashion. Keeping in mind that Microsoft recommends the entire conversation happen in less than 50 ms in general, if the DNS servers are taking more of that 50 ms than the actual LDAP search, this could be problematic. Timely is also subjective to whatever is normal for the environment and base lining is helpful here. In general if the DNS server is on the same segment and performing properly, 1 to 2 ms or less is reasonable. Of course as always, it is recommended that the environment be base lined so that the expected performance is known.
  • Use the network trace to determine if the TCP session with the GC is being established in a timely fashion.
    Again, keeping in mind that there is a 50 ms performance threshold, if the TCP session establishment is taking more time than the actual LDAP search takes to complete, this could be problematic. Also, timely is also subjective to whatever is normal for the environment and base lining is helpful hear. In general, if the GCs are on the same segments and both the network and the GC are performing properly, sessions established in 1 to 2 ms or less is reasonable. Of course as always, it is recommended that the environment be base lined so that the expected performance is known.
• AD Issues - Often AD infrastructures are large and the AD admins cannot determine which of the many GCs is in use by the affected Exchange server, thus the responsibility of pointing the AD support team to the right box to troubleshoot falls on the Exchange admin.
  • Use the network trace to find out which GC is having a large delay between the LDAP Request and LDAP Response packets.
  • Use the performance counters to narrow down to the box that is performing poorly.
    • Exchange 2007/2010 - MSExchange ADAccess Domain Controllers\LDAP Search Time and MSExchange ADAccess Domain Controllers\LDAP Read Time check the specific instances for values exceeding the recommended thresholds. This will report the specific GC (and underlying network path) that is not responding to Exchange as quickly as desired.
      Note: On Exchange 2007 SP2 and newer systems, the "MSExchange ADAccess Local Site Domain Controllers" can also be used in the same manner.
    • Exchange 2003 - MSExchangeDSAccess Domain Controllers\LDAP Search Time and MSExchangeDSAccess Domain Controllers\LDAP Read Time, check the specific instances for values exceeding the recommended thresholds. This will report the specific GC (and underlying network path) that is not responding to Exchange as quickly as desired.
    • Exchange 2000 - performance counters similar to the above did not exist. The only way to determine which GC is responding slowly is to analyze a network trace as described previously.

Next, identify bottlenecks within the GC (note: for the most part these basics apply to troubleshooting DNS performance as well):

While there are a large variety of potential issues, there are some basic things that can be eliminated before advanced troubleshooting need take place. The below guidance are the most basic remediation suggestions for the largest variety of these types of issues. Essentially this comes down to, "Ok, so the issue is identified, now what do I do?"

Since there is plenty of published information on how to isolate disk, processor, and/or network issues, please reference the excellent articles linked at the beginning for the recommended thresholds. Also, check out Performance Analysis of Logs (PAL), which automates the analysis of the performance data collected to help determine where the system is bottlenecking.

Disk bound - Once the issue is isolated to being disk bound, there are essentially two options, add memory or allow for increased disk IO (i.e. add spindles for direct attached storage).

• Adding memory is not quite as cut and dry as just throwing in some sticks of RAM.
  • 32-bit systems - It makes sense to avoid rehashing the entire discussion of 32-bit OS architecture especially since 32-bit architecture is on its way out. Reference this support article for some useful info. Memory usage by the Lsass.exe process on domain controllers that are running Windows Server 2003 or Windows 2000 Server
  • 64-bit systems - Generally the easiest thing to do is to put enough RAM in the box to load the entire database (minus white space) in memory. As of this writing, even for the largest enterprises (16 GB to 32 GB databases), this is becoming quite inexpensive or easy to justify. If the box has enough memory to load the entire database already and disk IO is the issue, troubleshoot why the database is not being loaded. A couple tip:
    • Measure Process\Virtual Bytes\lsass to see if it is about (within 10%/20%) of the size of the database (minus white space) or larger. If it isn't, troubleshoot why lsass isn't growing.
      Ensure that other applications aren't consuming large amounts of memory and preventing LSASS from growing like it wants to.
• Increasing disk-throughput.
  • If RAM cannot be added (or additional RAM will provide no benefit) due to the limits of the 32-bit OS architecture migrate to 64-bit. If migrating to 64-bit is not an option, the only option left is to add more disk throughput. Assuming that the storage is local and not SAN attached, it essentially means add more spindles (talk to the SAN guys if the AD database is stored on the SAN).
  • Reducing disk IO to the database volume is another option. Look at options such as turning off backups and AV scans during production hours, move SYSVOL and/or the database logs to separate spindles. Just follow standard storage performance tuning recommendations. Also, IOs to the disk can be reduced by reducing the demand on the boxes either by adding more DCs to the environment or reducing client demand. SPA and Performance Logs and Alerts will help track down what else is using the volume.

Processor bound:

• Ensure that LSASS is the process using the CPU, if not eliminate the problematic process.
• If LSASS is running hot, then use SPA (Windows 2003) or Performance Logs and Alerts (Windows 2008 and up) to identify where the CPU is being consumed within LSASS.
• If everything looks normal then the only option remaining is to add more processors to the box or more boxes to the infrastructure if there is high processor load across multiple systems.
  Note: In large infrastructures with multiple DCs and multiple Exchange servers, adding more hardware only makes sense if all of the boxes are experiencing high CPU utilization (close to or exceeding the thresholds in Ruling Out Active Directory-Bound Problems). If most or all of the boxes in the Exchange site are not experiencing high CPU utilization then there is probably a system specific reason for high CPU time. Look for hardcoded applications, improperly balanced load due to tuning LDAP Weights and Priorities (Check out Why you need Active Directory for Exchange Server 2007 for details on how this works), or anything else which may indicate why on GC is being singled out compared to the rest. Fix that issue before adding more hardware.

Network:

• If the outbound queue length is > 0 sustained the easy fixes are:
  • Update the NIC drivers to the latest versions if possible
  • Disable the Scalable Networking pack features (SNP) on Windows 2003 base servers- An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers. Check the network card settings to see if your driver can enable or disable certain SNP offloading features. Depending on the driver, if you disable these settings at the OS level, there is a possibility that they can still be enabled directly on the network card settings.
  • If the DCs are running Windows 2003 and the Exchange Servers are running on Windows 2008, then apply the Scalable Networking Pack rollup on the DCs to ensure that known compatibility issues do not occur. Also be aware of a known Windows 2008 TCP/IP setting issue in http://support.microsoft.com/kb/967224 that can affect overall network connectivity and performance.
  • Ensure that the NIC speed/duplex settings are properly set. MS recommendations, as of this writing, recommend both the GC and switch be set to 100/Full for 100 Mb segments and Auto/Auto for GigE segments. Check the Network Interface\Current Bandwidth performance counter to ensure that network card auto-negotiation is not changing the speeds the network card. This value should remain consistent at one value throughout the lifetime of the server being online.
  • Disable the load balancing/fault tolerance features of the NIC driver suite.
• Ensure that Network Interface\Packets Outbound Errors performance counters remains at 0 all the time. Any increases in this value can indicate intermittent connectivity issues on the server.
• If the network connection is using more 60% to 80% of the maximum bandwidth, the only solutions are to upgrade the available bandwidth (100Mb > GigE) or distribute the load across multiple servers/network connections.

Identifying the source of the GC load

The possibility exists that, despite the DC being either disk or processor bound, that the load is unnecessary. To determine this, the load on the box must be analyzed in greater detail than just the performance can provide. To that end, there are two main strategies for identifying the load on the box, both of which can be used concurrently. How to analyze this data is somewhat out of scope of this article, but the below bullets contain links to useful articles on how to use this data:

• Enable diagnostic logging on the DC by setting the "15 Field Engineering" registry value in order to identify expensive queries that return greater than 1,000 objects.
  • For more details on how to use this reference:
    • Logging LDAP searches: AD and ADAM > ActiveDir.org
    • Creating More Efficient Microsoft Active Directory-Enabled Applications
  • Pros
    • Shows the expensive or inefficient LDAP queries
    • The data is easy to get to since it is stored in the DS log
    • Data collection and alerting can be automated using SCOM.
  • Cons
    • Can overwrite the logs quickly, but the irony here is that if it is flushing the logs quickly, there is probably a lot of expensive queries that need to be remediated.
    • Doesn't actually track the processor impact of the query.
    • Only reports on LDAP queries.
• Use SPA (note this functionality exists natively in Windows 2008 and is entitled Performance Logs and Alerts which is part of Windows Reliability and Performance Monitor)
  • More details
    • Brad Rutkowski's Blog : Great tool for Windows 2003: Server Performance Advisor (SPA)
    • Download details: Microsoft ® Windows Server T 2003 Performance Advisor
    • Brad Rutkowski's Blog : Interacting with Data Collector Sets via Powershell
    • tonymurray : Troubleshooting LDAP issues with Server Performance Advisor (SPA)
    • Tracking LDAP Searches with Windows Server 2008 Reliability and Performance Monitor > ActiveDir.org
  • Pros
    • Shows everything that is going on, whether it is authentication, LDAP queries, network traffic, or disk IO.
    • Data collection can be triggered using SCOM when certain thresholds are hit.
  • Cons
    • On busy DCs the logging can quickly grow to unmanageable levels (busy being hundreds of LDAP queries and authentications per second). Think on the order of 100+ MB per minute. But this isn't too bad as usually a 5 to 10 minute data collection during a production issue will get the data necessary to identify what is occurring.
    • Requires the installation of a tool set (Windows 2003 only)

Conclusion

Data collection is key! Getting the right data when the problem occurs is absolutely critical to the restoring the health of the environment. Furthermore, as Exchange is the product suffering when its dependencies are not responding adequately, it falls on the Exchange administrator to do the leg work collecting the right data from the perspective of Exchange, and then isolating where the network, DNS, and AD administrators might need to look and start collecting data. Without this information, the network, DNS, and AD admins will be looking for a needle in a haystack and it is highly unlikely that they will find anything of use. Thus, once the Exchange admin has isolated the dependencies of concern, using the information provided here, it will be much easier to collect the data necessary from those systems to remediate any issues.

Thanks to Rod Fournier and Mike Lagase for their endless suggestions for improvement of this blog post.

- Ken Brumfield