You Had Me At EHLO...

Exchange Scalability Limits Worksheet Available

2010-03-12T21:00:00Z

No matter how many lines of code it has, every software application has scalability limits.

Some limits can be caused by limits that are external to the application itself, while others can be inherent within the application. As a software engineering team, it is important for us to understand the scalability limits of the solution as a whole; this includes the limits of both external dependencies and internal code. Not only does this enable us to document and communicate this information to our customers, but it also helps us make better software. It provides us with a deeper understanding of our application within the most demanding environments.

Exchange 2010 is new and different from its predecessors in that, unlike all previous versions of Exchange, Exchange 2010 is the first version designed specifically to run as an on-premises application, as a hosted service in the cloud, and as a combination of both (cross-premise). At 20 million mailboxes and growing, it's more important than ever that our understanding of the limits of Exchange be crisp.

A while ago, we began the process of documenting the scalability limits of both Exchange 2007 SP2 and Exchange 2010. We've looked at all areas: things like organization sizing, rules sizing, IRM-protected messages, secure channels, named properties, load balancing, network latency, and more.

Here's our initial version of Exchange Scale Limitations spreadsheet documenting our findings thus far. There's a lot of great information in there, including issue descriptions, mitigations, and links where appropriate.

We hope you enjoy this first version. Please feel free to send us your comments and feedback.

-- Erin Bookey

GAL Photos in Exchange 2010 and Outlook 2010

2010-03-11T00:51:00Z

Over the years, displaying recipient photographs in the Global Address List (GAL) has been a frequently-requested feature, high on the wish lists of many Exchange folks. Particularly in large organizations or geographically dispersed teams, it's great to be able to put a face to a name for people you've never met or don't frequently have face time with. Employees are commonly photographed when issuing badges/IDs, and many organizations publish the photos on intranets.

There have been questions about workarounds or third-party add-ins for Outlook, and you can also find some sample code on MSDN and elsewhere. A few years ago, an IT person wrote ASP code to make employee photos show up on the intranet based on the Employee ID attribute in Active Directory (AD) - which was imported from the company's LDAP directory. A fun project to satisfy the coder alter-ego of the IT person.

Luckily, you won't need to turn to your alter-ego to do this. Exchange 2010 and Outlook 2010 make this task a snap, with help from AD. AD includes the Picture attribute (we'll refer to it using its ldapDisplayName: thumbnailPhoto) to store thumbnail photos, and you can easily import photos— not the high-res ones from your 20 megapixel digital camera, but small, less-than-10K-ish ones, using Exchange 2010's Import-RecipientDataProperty cmdlet.

The first question most IT folks would want to ask is— What's importing all those photos going to do to the size of my AD database? And how much AD replication traffic will this generate? The attribute is limited to 10K in size, and the cmdlet won't allow you to import a photo that's larger than 10K. The original picture used in this example was 9K, and you can compress it further to a much smaller size - let's say approximately 2K-2.5K, without any noticeable degradation when displayed at the smaller sizes. If you store user certificates in AD, the 10K or smaller size thumbnail pictures are comparable in size. Storing thumbnails for 10,000 users would take close to 100 Mb, and it's data that doesn't change frequently.

Note: The recommended thumbnail photo size in pixels is 96x96 pixels.

With that out of the way, let's go through the process of adding pictures.

A minor schema change

First stop, the AD Schema. A minor schema modification is required to flip the thumbnailPhoto attribute to make it replicate to the Global Catalog.

If you haven't registered the Schema MMC snap-in on the server you want to make this change on, go ahead and do so using the following command:
Regsvr32 schmmgmt.dll
Fire up a MMC console (Start -> Run -> MMC) and add the Schema snap-in
In the Active Directory Schema snap-in, expand the Attributes node, and then locate the thumbnailPhoto attribute. (The Schema snap-in lists attributes by its ldapDisplayName).
In the Properties page, select Replicate this attribute to the Global Catalog, and click OK.

Figure 1: Modifying the thumbnailPhoto attribute to replicate it to Global Catalog

Loading pictures into Active Directory

Now you can start uploading pictures to Active Directory using the Import-RecipientDataProperty cmdlet, as shown in this example:

Import-RecipientDataProperty -Identity "Bharat Suneja" -Picture -FileData ([Byte[]]$(Get-Content -Path "C:\pictures\BharatSuneja.jpg" -Encoding Byte -ReadCount 0))

To perform a bulk operation you can use Get-Mailbox cmdlet with your choice of filter (or Get-DistributionGroupMember if you want to do this for members of a distribution group), and pipe the mailboxes to a foreach loop. You can also retrieve the user name and path to the thumbnail picture from a CSV/TXT file.

Thumbnails in Outlook 2010

Now, let's fire up Outlook 2010 and take a look what that looks like.

In the Address Book/GAL properties for the recipient

Figure 2: Thumbnail displayed in a recipient's property pages in the GAL

When you receive a message from a user who has the thumbnail populated, it shows up in the message preview.

Figure 3: Thumbnail displayed in a message

While composing a message, the thumbnail also shows up when you hover the mouse on the recipient's name.

Figure 4: Recipient's thumbnail displayed on mouse over when composing a message

There are other locations in Outlook where photos are displayed. For example, in the Account Settings section in the Backstage Help view.

Update from the Outlook team

Our friends from the Outlook team have requested us to point out that the new Outlook Social Connector also displays GAL Photos, as well as photos from Contacts folders and from social networks, as shown in this screenshot.

Figure 5: Thumbnail photos displayed in the People Pane in the Outlook Social Connector

More details and video in Announcing the Outlook Social Connector on the Outlook team blog.

GAL Photos and the Offline Address Book

After you've loaded photos in Active Directory, you'll need to update the Offline Address Book (OAB) for Outlook cached mode clients. This example updates the Default Offline Address Book:

Update-OfflineAddressBook "Default Offline Address Book"

In Exchange 2010, the attributes in an OAB can be customized. This is done using the ConfiguredAttributes property of the OAB (see Set-OfflineAddressBook cmdlet). ConfiguredAttributes is populated with the default set of attributes, and you can modify it to add/remove attributes as required.

By default, thumbnailPhoto is included in the OAB as an Indicator attribute. This means the value of the attribute isn't copied to the OAB— instead, it simply indicates the client should get the value from AD. If an Outlook client (including Outlook Anywhere clients connected to Exchange using HTTPS) can access AD, the thumbnail will be downloaded and displayed. When offline, no thumbnail downloads. Another example of an Indicator attribute is the UmSpokenName.

You can list all attributes included in the default OAB using the following command:

(Get-OfflineAddressBook "Default Offline Address Book").ConfiguredAttributes

For true offline use, you could modify the ConfiguredAttributes of an OAB to make thumbnailPhoto a Value attribute. After this is done and the OAB updated, the photos are added to the OAB (yes, all 20,000 photos you just uploaded...). Depending on the number of users and sizes of thumbnail photos uploaded, this would add significant bulk to the OAB. Test this scenario thoroughly in a lab environment— chances are you may not want to provide the GAL photo bliss to offline clients in this manner.

To prevent Outlook cached mode clients from displaying thumbnail photos (remember, the photo is not in the OAB— just a pointer to go fetch it from AD), you can remove the thumbnailPhoto attribute from the ConfiguredAttributes property of an OAB using the following command:

$attributes = (Get-OfflineAddressBook "Default Offline Address Book").ConfiguredAttributes
$attributes.Remove("thumbnailphoto,Indicator")
Set-OfflineAddressBook "Default Offline Address Book" -ConfiguredAttributes $attributes

-Bharat Suneja

Dude, where's my rollup?

2010-03-08T21:57:18Z

So I just installed RU1 on my brand new Exchange 2010 server and then I issue a Get-Exchangeserver -Identity MyExchangeServer and get the following output for AdminsDisplayVersion and ExchangeVersion:

Ok that looks a little familiar for some reason. I go to my Exchange 2010 RTM server and issue the same CMDlet and get:

...The same result! But one server has RU1 installed and the other is RTM. Shouldn't I get a different version number back?

Well... no. Exchange 2007 and forward do not reflect the version number either in the value for AdminDisplayVersion, ExchangeVersion, or at this registry key HKLM\SOFTWARE\Microsoft\v8.0\<Role>\ConfiguredVersion as influenced by roll ups. This is a common misconception.

The most conclusive way to get the version of your exchange server, rollup and all, is to check the file version of ExSetup.exe in the BIN folder.

Here is Exchange 2010 RU1 version:

And here is Exchange 2010 RTM:

Another way of getting this information is to run the following PowerShell one-liner:

GCM exsetup |%{$_.Fileversioninfo}

The below output is from an exchange 2010 server running RU1:

Here is an exchange 2010 RTM server:

You can then correlate the version number you find with those listed here, here or on the actual rollup update download pages.

Hope this post reduces some confusion out there!

- Tom Kern

Released: Update Rollup 2 for Exchange Server 2010

2010-03-05T19:58:00Z

Exchange CXP team has released Update Rollup 2 for Exchange Server 2010 RTM (KB 979611) to the download center.

KB 979611 lists all the fixes included in this rollup. Here are some of the product improvements and critical bug fixes we'd like to call out starting with a couple of IMAP improvements:

KB 977633 This fixes IMAP4 clients ability to log on to their mailboxes if the mailboxes are located on Exchange 2003 backend servers and if the clients are connecting via Exchange 2010 CAS servers.
KB 979480 IMAPid was not working correctly after moving a lot of users from one Exchange 2010 server to another*. IMAP4 users complained about the inbox not being updated any more. Old messages were still visible, but messages which were received after the mailbox move were not visible. The problem affected different IMAP Clients. The problem did not affect MAPI clients and OWA. Now it is fixed up. *(Specifically this occurred in the situation with same DAG, now local storage instead of iSCSI storage, all servers are Exchange 2010 with Update Rollup 1 installed on Windows Server 2008 R2).
KB 979431 When user migrated from Exchange Server 2003 to Exchange Server 2010, and that user connected via POP3, the POP3 service crashed. This was fixed up so it will not crash.
KB 979563 Push Notifications didn't work because Exchange Server 2010 was not sending SOAPAction header in the notify callback. This caused Exchange to receive a HTTP 500 response from the notification client and the webservice failed. Push notifications should now properly send that SOAP header.
KB 980261 We fixed passive page patching when diagnostic tracing code was needed for forensic analysis that was generating a -1022 error case.
KB 980262 Source side log copier errors are more gracefully handled when the log has a bad block and the read fails.
KB 979566 Activesync proxy was failing for linked mailboxes in a CAS to CAS proxy scenario where the users token is serialized and sent in the request. When attempting to create the client security context from the SID, a AuthZException was thrown because we did not have access to the token information of the linked account, so now for this it no longer throws exceptions.

Only the English Rollup?

Customers may not install the rollup because they may feel that this should only be installed on an English Exchange Server. This was true for Exchange 2007 and is not true for Exchange 2010. When installing this rollup, the UI will be English and the “Add/Remove Programs” text will be English. We are expecting to release the other rollup installation language strings with the next rollup. We are finishing UI validation.

Known Issue

With Update Rollup 2 for Exchange Server 2010 RTM, we introduced a new parameter for the Set-ClientAccessServer cmdlet, CleanUpInvalidAlternateServiceAccountCredentials. Unfornately, the parameter cannot be used at this time. However the Set-ClientAccessServer cmdlet still functions with all other available parameters.

The cmdlet functions but not the parameter because of how RBAC works at the Organization\Enterprise level. The change functions as expected, except for this one issue. This issue blocks some functionality offered by this particular fix. We have a work around and we are currently performing testing and ensuring that we document it correctly. It will require running “Install-CannedRbacRoles” on one server after the rollup is installed. Once replication happens across the AD, the parameter will be available for use for the servers that have the rollup installed.

KB 979611 has more details about this release and a complete list of all fixes included in this rollup.

-Exchange

Removing a Feature From The Exchange Control Panel

2010-03-04T19:00:00Z

For various reasons, there are times when an administrator does not want a part of the ECP to be accessible by some users and they desire a features' tab or entry point to not be visible at all. The web.config file for the Exchange Control Panel (ECP) contains the requirements a logged in user must meet before the feature tab or configuration item may be displayed in the UI.

Here we will step through an example of how to go through the process of determining what you must do to accomplish this task.

IMPORTANT: Exchange Control Panel files are not modified to accomplish this — the process only involves changing the user's RBAC Role assignment.

SUPPORT NOTE: Modifying the Exchange Control Panel files to remove parts of the UI is not supported. Serious problems may occur if you modify web.config files. The only supported way of removing a feature from the ECP is by modifying the effective rights a user has using RBAC, as documented in this post.

In Exchange 2010, Role Based Access Control (RBAC) is the new permissions model that allows you to assign granular permissions based on management roles. To learn more about RBAC, see Understanding Role Based Access Control in Exchange 2010 documentation, and the previous post RBAC and the Triangle of Power.

Remove the Delivery Reports tab for a user

In Exchange 2010, the Delivery Reports tab in ECP allows users to retrieve delivery reports for messages sent to or received by them. In this example, the goal is to not display the Delivery Reports tab in ECP so it's not accessible by a user.

Figure 1: The Delivery Reports tab in ECP

To remove the Delivery Reports tab from ECP for a user, we need to determine what's needed for the tab to show. To get this information, we need to check the Web.Config file located in ECP's folder at ":\Program Files\Microsoft\Exchange Server\V14\ClientAccess\ecp\Reporting". ECP uses the authorization section of the Web.Config file to evaluate if the tab should be displayed. If the user is not allowed to run the cmdlet shown, the tab is not displayed. Let's view the Authorization section of the Deliveryreports.slab location path:

<location path="DeliveryReports.slab">
       <system.web>
           <authorization>
                <allow roles="Search-MessageTrackingReport@R:Organization" />
                <!-Deny everyone else ->
                <deny users="*" />
            </authorization>

As shown in the above figure, access to the Search-MessageTrackingReport cmdlet is required to display the Delivery Reports tab. To disable the Delivery Reports tab, we need to determine which RBAC roles can run the Search-MessageTrackingReport cmdlet, so we can remove the permission for the user to run it. This ensures the tab will not be displayed to that user.

To determine which RBAC roles can run the Search-MessageTrackingReport cmdlet, we use the Get-ManagementRole cmdlet:

Get-ManagementRole -cmdlet Search-MessageTrackingReport
The result:

Name                   RoleType
-------                --------------
Message Tracking       MessageTracking
View-OnlyConfiguration ViewOnlyConfiguration
MyBaseOptions          MyBaseOptions

Next we must determine which of the above roles the user is a member of and where it would make sense to remove the Search-MessageTrackingReport cmdlet from. For example, we wouldn't want to remove the cmdlet from the ViewOnly Configuration because that is an administrative role. The user is not an administrator, and therefore it's not likely that he/she has been assigned the MessageTracking role. This means that we will have to check to see what roles/assignments the user is a member of:

Get-RoleGroup | where {$_.Members -like "*Display UserName*"} | fl name
The command doesn't return any results because the user is not a member of any administrator type role. Next, we will check the management role assignments for this user:
Get-ManagementRoleAssignment -RoleAssignee UserName

Among other items you see the list of roles (note these are user/self configuration roles):
Name                                                         Role
--------                                                     ---------
MyBaseOptions-Default Role Assignment Policy                 MyBaseOptions
MyContactInformation-Default Role Assignment Policy          MyContactInformation
MyVoiceMail-Default Role Assignment Policy                   MyVoiceMail
MyDistributionGroupMembership-Default Role Assignment Policy MyDistributionGroupMembership
Custom Default Policy                                        MyDiagnostics

It looks like the only one we are interested in here is the "MyBaseOptions" because we already know that the cmdlet we want to block is only available in that role that the user has anything to do with. The user is not an administrator so the other roles are not interesting to us for this scenario.

To make sure the user is assigned to the role assignment policy we can verify:

Get-Mailbox UserName | fl roleassignmentpolicy

RoleAssignmentPolicy: Default Role Assignment Policy
Tip: If you want to combine some of the above steps into one line to find out which role contains that cmdlet we are interested in (Search-MessageTrackingReport), you can use the following set of cmdlets:

Get-ManagementRole -Cmdlet Search-MessageTrackingReport | Get-ManagementRoleAssignment -RoleAssignee UserName -Delegating $False | FT Role, RoleAssigneeName

The result:

Role            RoleAssigneeName
----            ----------------
MyBaseOptions   Default Role Assignment Policy
Now, we know that we need to create a new Role Assignment Policy for the user and associate it with a new (customized) MyBaseOptions role. We will make a copy of the MyBaseOptions role so we can remove the Search-MessageTrackingReport cmdlet from it.

First, we will create a new (end user) Role Assignment Policy called Alternate Assignment Policy, and leave the original policy unchanged (for other users who should still have access to the Delivery Reports tab).:

New-RoleAssignmentPolicy "Alternate Assignment Policy"
For this new policy, we need to turn on a few of the default options that the Default Policy had. For example, to add the ability for the user to edit their own contact information we add the MyContactInformation role to the policy:

New-ManagementRoleAssignment -Name "MyContactInformation-Alternate Assignment Policy" -policy "Alternate Assignment Policy" - role MyContactInformation

To add the ability for the user to manage their own distribution group membership, we add the MyDistributionGroupMembership role to the policy:

New-ManagementRoleAssignment -Name "MyDistributionGroupMembership-Alternate Assignment Policy" -policy "Alternate Assignment Policy" - role MyDistributionGroupMembership
Now we need to create a copy of the MyBaseOptions role so we can remove the Search-MessageTrackingReport cmdlet from it and then assign it to the new Role Assignment Policy. We can give it any name, preferably something with a good description.:

New-ManagementRole "MyBaseOptionsWithoutMessageTracking" -Parent MyBaseOptions
We remove the Search-MessageTrackingReport cmdlet from the "MyBaseOptionsWithoutMessageTracking" role:
Remove-ManagementRoleEntry "MyBaseOptionsWithoutMessageTracking\Search-MessageTrackingReport"
Next, we assign the newly created MyBaseOptionsWithoutMessageTracking role to the Role Assignment Policy:
New-ManagementRoleAssignment -Name "MyBaseOptionsWithoutMessageTracking-Alternate Assignment Policy" -policy "Alternate Assignment Policy" - role MyBaseOptionsWithoutMessageTracking
Then, we assign the Role Assignment Policy to the user:
Set-mailbox mod1user1 -RoleAssignmentPolicy "Alternate Assignment Policy"

This can also be performed in the ECP, as shown in figure 2.

Figure 2: Assigning the Role Assignment Policy to the user in ECP

Done! Now we can test the user experience. As shown in figure 3, when UserName logs on, the Delivery Reports tab isn't visible.

Figure 3: The Delivery Reports tab is removed for the user

After the Delivery Reports tab is removed, if your user tries to track a message from within Outlook Web App or Outlook, he/she will receive the following error:

Figure 4: Error when user tries to track a message

-Perry Newman

Exchange Server in Depth: Master Concepts and Optimize Your Server Environment

2010-02-27T16:06:00Z

The latest release of Microsoft Exchange can help you achieve better business outcomes while controlling the costs of deployment, administration, and compliance. Exchange 2010 delivers a wide range of deployment options, integrated information leakage protection, and advanced compliance capabilities. Check out our free webcasts and podcasts, and then try Exchange 2010 for yourself by taking one of our virtual labs or labcasts.

TechNet Webcast: Microsoft Exchange Server 2010 Management and Operations (Level 200)
Thursday, March 11, 2010
11:00 A.M.-12:30 P.M. Pacific Time
TechNet Webcast: Microsoft Exchange Server 2010 Unified Messaging (Level 300)
Tuesday, March 16, 2010
11:00 A.M.-12:30 P.M. Pacific Time
TechNet Webcast: Calendar Sharing and Federation in Microsoft Exchange Server 2010 (Level 300)
Thursday, March 18, 2010
11:00 A.M.-12:30 P.M. Pacific Time
TechNet Webcast: Deploying and Managing Microsoft Exchange Server 2010 Transport Servers (Level 200)
Tuesday, March 23, 2010
11:00 A.M.-12:30 P.M. Pacific Time
TechNet Webcast: Microsoft Exchange Server 2010 Storage Architecture (Level 300)
Thursday, March 25, 2010
11:00 A.M.-12:30 P.M. Pacific Time

Announcing the Release of the Exchange Server Pre-Deployment Analyzer

2010-02-24T20:33:00Z

Today I am pleased to announce the release of the Exchange Server Pre-Deployment Analyzer (ExPDA).

You can use the Exchange Pre-Deployment Analyzer to perform an overall topology readiness scan of your environment. When you run the Exchange Pre-Deployment Analyzer, it provides a detailed report that will alert you if there are any issues within your organization, which could prevent you from deploying Exchange 2010. For example, the Exchange Pre-Deployment Analyzer will notify you if you haven't deployed the minimum required Exchange service pack on all your existing Exchange servers.

The checks performed by ExPDA are similar to the pre-requisite checks implemented (via Exchange Best Practices Analyzer) in the Exchange 2010 Setup program; in fact ExPDA is based off the Exchange Best Practices Analyzer (ExBPA) engine. However, unlike Exchange 2010 setup, this tool focuses only on overall topology readiness and not the ability to run Exchange 2010 on the local computer. The scan also performs a deep analysis of each existing Exchange 2003/2007 server to verify that it has the necessary updates and configuration in-place to support Exchange 2010. The end report is structured as follows:

Critical - A configuration problem that will prevent Exchange 2010 from being deployed in the organization. For example, the Active Directory Forest is not operating in Windows Server 2003 Forest Functional Mode or higher.
Warning - A configuration item that may prevent customers having the best possible experience with Exchange 2010. A warning may also reflect some functionality that is not available in Exchange 2010.

ExPDA is another component in our vision to provide a seamless upgrade experience that reduces the complexities in deploying Exchange 2010. To start planning your upgrade, please utilize the Exchange Deployment Assistant. The Deployment Assistant allows a customer to create Exchange 2010 on-premises deployment instructions that are customized to their environment. The Assistant asks a small set of questions, and based on the answers, it provides a finite set of instructions that are designed to get a customer up and running on Exchange 2010. Running the Exchange Server Pre-Deployment Analyzer is now a recommended step within the pre-requisites section of the Deployment Assistant.

You can download ExPDA at http://www.microsoft.com/downloads/details.aspx?FamilyID=88b304e7-9912-4cb0-8ead-7479dab1abf2&displaylang=en. ExPDA is supported on Windows 7, Windows Vista with Service Pack 2, Windows Server 2008 with Service Pack 2, Windows Server 2008 R2, and Windows Server 2003 with Service Pack 2.

Q&A

Does this tool replace the Exchange Deployment Assistant?
No, ExPDA is merely an additional tool that can be used as a step within the upgrade experience. The Exchange Deployment Assistant will walk you through all aspects of the upgrade, namely how to coexist properly with Exchange 2010 and legacy versions of Exchange, whereas, ExPDA is one step within that process and ensures that the environment is ready to have the first Exchange 2010 server deployed.
I ran the scan and have questions about the results. What should I do?
If you'd like to read more about the requirements of Exchange 2010, please see the Planning for Exchange 2010 section on TechNet.
I ran the scan and received unexpected results. If I think there's a bug, who can I contact?
If you need assistance, please visit the Exchange Server Deployment Forum or you can send mail to exbpafb AT Microsoft DOT com.
Is this new functionality available in all languages?
No. ExPDA is only available as a U.S. English version.
I'd like to know if my organization is capable of running Exchange 2007. Can I use ExPDA to check this?
No, ExPDA only verifies if an organization is ready to have the first Exchange 2010 server installed. If you need to determine whether your organization is ready to have Exchange 2007 deployed, you can utilize ExBPA v2.8 and the latest Exchange 2007 ExBPA.Readiness.xml:
1. Install ExBPA 2.8 and the last ExBPA 2.8 Update.
2. Download the latest Exchange 2007 service pack rollup. At the time of this writing this is SP2 RU2.
3. Extract the rollup binaries using this command: msiexec /a filepath to MSI file /qb TARGETDIR=filepath to target folder
4. Copy the ExBPA.Readiness.xml to the \en folder.
5. Launch ExBPA.
Where can I look at the list of checks made in this new scan?
If you're familiar with the BPA XML files, the new check is wholly contained within the ExBPA.Readiness.xml file located within %Program Files%\Microsoft\Exchange Server\V14\ExPDA\en folder.
- Reporting the number of Active Directory trees, domains, sites, admin groups, routing groups, Exchange 5.5 servers, Exchange 2000 servers, Exchange 2003 servers, total mailboxes, Windows 2000 Active Directory servers, Windows Server Active Directory servers, Windows Server Active Directory servers. Report how many Active Directory domain/sites have Exchange servers installed.
- Verifying that the Schema Master is Windows 2003 SP1 or later.
- Identifying Active Directory domains that are not in native mode.
- Identifying Active Directory sites that do not have a global catalog server running Windows 2003 SP1 or later.
- Verifying that there are zero Active Directory Connector servers in existence.
- Identifying any SMTP site links in existence.
- Verifying that the Exchange organization is in native mode.
- Identifying any non-standard proxy address generators.
- Identifying whether you have any ambiguously defined email addresses in your recipient policies.
- Identifying any non-MAPI public folder hierarchies (a.k.a. AppTLH's) in use.
- Identifying Routing Groups that span Active Directory sites.
- Identifying any Active Directory sites that span Routing Groups.
- Identifying any Routing Group Connectors that have specialized settings (activation, max size, accept/reject lists, restrict message type/priority).
- Identifying any SMTP Connectors that support non-SMTP address spaces.
- Identifying any SMTP Connectors that use inline domain-wildcarding for address spaces (e.g. *foo.com instead of *.foo.com).
- Identifying any X.400 Connectors in the topology.
- Identifying any EDK-based Connectors in the topology (excluding Notes).
- Verifying that any servers running Exchange 2003 have SP2 or later.
- Verifying that any servers running Exchange 2007 have SP2 or later.
- Identifying any SMTP virtual servers that are not using port 25 for incoming/outgoing.
- Verifying that all Exchange 2003 servers have SuppressStateChanges set.
- Identifying any Exchange 2003 servers that have active NNTP newsfeeds.
- Identifying any Exchange 2003 servers that use the Event Scripting service.
- Identifying any Exchange 2003 servers that have the ExIFS (a.k.a. M:) drive enabled.
- Identifying any parts of Active Directory that have Access Control Entry inheritance disabled.

Thanks for your continued support. Feel free to post feedback in comments of this blog post, or visit the Exchange Server Deployment Forum.

- Ross Smith IV

Dude, Where's My Single Instance?

2010-02-22T15:49:05Z

In Exchange Server 2010, there is no more single instance storage (SIS). To help understand why SIS is gone, let's review a brief history of Exchange.

During the development of Exchange 4.0, we had two primary goals in mind, and SIS was borne out of these goals:

Ensure that messages were delivered as fast and as efficient as possible.
Reduce the amount of disk space required to store messages, as disk capacity was premium.

Exchange 4.0 (and, to a certain extent, Exchange 5.0 and Exchange 5.5) was really designed as a departmental solution. Back then, users were typically placed on an Exchange server based on their organization structure (often, the entire company was on the same server). Since there was only one mailbox database, we maximized our use of SIS for both message delivery (only store the body and attachments once) and space efficiency. The only time we created another copy within the store was when the user modified their individual instance.

For almost 19 years, the internal Exchange database table structure has remained relatively the same:

Then came Exchange 2000. In Exchange 2000, we evolved considerably - we moved to SMTP for server-to-server connectivity, we added storage groups, and we increased the maximum number of databases per server. The result was a shift away from a departmental usage of Exchange to enterprise usage of Exchange. Moreover, the move to 20 databases reduced SIS effects on space efficiency, as the likelihood that multiple recipients were on the same database decreased. Similarly, message delivery was improved by our optimizations in transport, so transport no longer benefited as much from SIS either.

With Exchange 2003, consolidation of servers took off in earnest due to features like Cached Exchange Mode. Again the move away from departmental usage continued. Many customers moved away from distributing mailboxes based on their organization structure to randomization of the user population across all databases in the organization. Once again, the space efficiency effects of SIS were further reduced.

In Exchange 2007, we increased the number of databases you could deploy, which again reduced the space efficiency of SIS. We further optimized transport delivery and completely removed the need for SIS from a transport perspective. Finally, we made changes to the information store that removed the ability to single instance message bodies (but allowed single instancing of attachments). The result was that SIS no longer provided any real space savings - typically only about 0-20%.

One of our main goals for Exchange 2010 was to provide very large mailboxes at a low cost. Disk capacity is no longer a premium; disk space is very inexpensive and IT shops can take advantage of larger, cheaper disks to reduce their overall cost. In order to leverage those larger capacity disks, you also need to increase mailbox sizes (and remove PSTs and leverage the personal archive and records management capabilities) so that you can ensure that you are designing your storage to be both IO efficient and capacity efficient.

During the development of Exchange 2010, we realized that having a table structure optimized for SIS was holding us back from making the storage innovations that were necessary to achieve our goals. In order to improve the store and ESE, to change our IO profile (from many, small, random IOs to larger, fewer, more sequential IOs), and to resolve our inefficiencies around item count, we had to change the store schema. Specifically, we moved away from a per-database table structure to a per-mailbox table structure:

This architecture, along with other changes to the ESE and store engines (lazy view updates, space hints, page size increase, b+ tree defrag, etc.), netted us not only a 70% reduction in IO over Exchange 2007, but also substantially increased our ability to store more items in critical path folders.

As a result of the new architecture and the other changes to the store and ESE, we had to deal with an unintended side effect. While these changes greatly improved our IO efficiency, they made our space efficiency worse. In fact, on average they increased the size of the Exchange database by about 20% over Exchange 2007. To overcome this bloating effect, we implemented a targeted compression mechanism (using either 7-bit or XPRESS, which is the Microsoft implementation of the LZ77 algorithm) that specifically compresses message headers and bodies that are either text or HTML-based (attachments are not compressed as typically they exist in their most compressed state already). The result of this work is that we see database sizes on par with Exchange 2007.

The below graph shows a comparison of database sizes for Exchange 2007 and Exchange 2010 with different types of message data:

As you can see, Exchange 2007 databases that contained 100% Rich Text Format (RTF) content was our baseline goal when implementing database compression in Exchange 2010. What we found is that with a mix of messaging data (77% HTML, 15% RTF, 8% Text, with an average message size of 50KB) that our compression algorithms are on par with Exchange 2007 database sizes. In other words, we mitigated most of the bloat caused by the lack of SIS.

Is compression the answer to replacing single instancing all together? The answer to that question is that it really does depend. There are certain scenarios where SIS may be viable:

Environments that only send Rich-Text Format messages. The compression algorithms in Exchange 2010 do not compress RTF message blobs because they already exist in their most compressible form.
Sending large attachments to many users. For example, sending a large (30 MB+) attachment to 20 users. Even if there were only 5 recipients out of the 20 on the same database, in Exchange 2003 that meant the 30MB attachment was stored once instead of 5 times on that database. In Exchange 2010, that attachment is stored 5 times (150 MB for that database) and isn't compressed. But depending on your storage architecture, the capacity to handle this should be there. Also, your retention requirements will help here, by forcing the removal of the data after a certain period of time.
Business or organizational archives that are used to maintain immutable copies of messaging data benefit from single instancing because the system only has to keep one copy of the data, which is useful when you need to maintain that data indefinitely for compliance purposes.

If you go back through our guidance over the past 10 years, you will never find a single reference to using SIS around capacity planning. We might mention it has an impact in terms of the database size, but that's it. All of our guidance has always dictated designing the storage without SIS in mind. And for those that are thinking about thin provisioning, SIS isn't a reason to do thin provisioning, nor is SIS a means to calculate your space requirements. Thin provisioning requires an operational maturity that can react quickly to changes in the messaging environment, as well as, a deep understanding of the how the user population behaves and grows over time to sufficiently allocate the right amount of storage upfront.

In summary, Exchange 2010 changes the messaging landscape. The architectural changes we have implemented enable the commoditization of email - providing very large mailboxes at a low cost. Disk capacity is no longer a premium. Disk space is cheap and IT shops can take advantage of larger, cheaper disks to reduce their overall cost. With Exchange 2010 you can deploy a highly available system with a degree of storage efficiency without SIS at a fraction of the cost that was required with previous versions of Exchange.

So, there you have it. SIS is gone.

- Ross Smith IV

E2010 Mailbox Server Role Requirements Calculator v4.5

2010-02-17T16:48:19Z

Please go to our Mailbox Server Role Storage Requirements Calculator updates tracking page to see what is in this new version!

A blog post explaining the calculator (updated for this new version) is here.

Comments welcome!

- Ross Smith IV

Call Answering Rules (Part 2): Call Answering Conditions

2010-02-16T23:04:00Z

In Exchange 2010, users can easily create call answering rules to have a call answered based on the specified conditions. Users can create rules to answer calls from specified callers or Contacts in a certain way, handle calls differently based on their availability (Free/Busy status) or time of day. The call answering rule will only be triggered when all the associated conditions are met. In this post, I will cover:

The different classes of conditions supported in UM 2010
How does UM go about evaluating a given set of call answering rules

This article assumes that you have read my previous post Call Answering Rules (Part I) - Creating your first Call Answering Rule.

The Call Answering Rule form

The diagram below shows you the Call Answering Rule form. In particular:

Condition(s) which have already been added to the rule is located on the left (in green). You can remove them by clicking on the symbol beside the condition.
Condition(s) which you can further add to this rule is located on the right (in red). To start adding new condition, simplify click on the condition desired.

Figure 1: The call answering rule form

Different classes of conditions

There are 4 different classes of conditions supported by UM 2010, namely:

Caller identity
Time-of-the-day
Calendar free/busy status
When my automated email reply is turned on

Caller Identity
Users can use different call answering rules for different callers. For example, if you receive a call from an important customer's phone number, you can answer it differently. To answer calls based on caller identity, users can add a caller-ID condition to a call answering rule such that the rule will only be triggered if the calling party matches one of those specified. To configure this condition:
1. Click on "If the caller is." condition to bring up the following UI control.
  
  Figure 2: Call answering rule based on caller
2. Specify 1 or more phone numbers, pick Contacts from the GAL or your Contacts folder, or specify the entire Contacts folder.
3. Click on "Apply" to close the UI control.
Time of Day
Users can add a time-of-the-day condition to a call answering rule such that the rule will only be triggered if the time of the call matches the time period specified. For example, during working hours, calls can be answered using the "Follow-Me" action, or routed straight to voicemail during after-hours. To configure this condition:
1. Click on If it is during this period. condition to bring up the following UI control:
  
  Figure 3: Call answering rule based on time of day
2. Specify a period, either working hours, non-working hours or custom hours.
3. Click on Apply to close the UI control.
Calendar Free/Busy status

Users can specify a condition for a call answering rule based on their availability (calendar Free/Busy status). To do so:
1. Click on If my schedule shows that my status is. condition to bring up the following UI control:
  
  Figure 4: Call answering rule based on Free/Busy Status
2. Pick one or more Free/Busy status which you want this rule to fire. In the example provided, I have specified that the call answering rule to be triggered if my calendar shows that I am either Busy or Away.
3. Click on Apply to close the UI control.
When my automated email reply is turned on

Users can specify that a call answering rule only be fired when their automatic email reply is turned on. To do so, simply click on "If Automatic Replies are turned on" condition.
Call Answering Rules and conditions

It is possible to associate the same call answering rule with multiple conditions. All the conditions must be met in order for the call answering rule to be trigger. You can also create call answering rules without specifying any condition. When such a rule is encountered, it will be triggered as though all its conditions are met.
Multiple Call Answering Rules

You can create one or more call answering rules. You can also associate each call answering rule with one or more conditions. The diagram below illustrates an example with 3 call answering rules configured. You can enable/disable or reorder the rules.

Figure 5: Multiple call answering rules

As illustrated in the flow diagram below, for each call answering call received:
- No call answering rules configured: If you do not have any call answering rules configured, UM will offer the caller to leave a voice message.
- One or more call answering rules configured: If there are one or more call answering rules configured, UM will evaluate these rules in a top-down fashion. The first rule, whose conditions are met, will be fired.
- After evaluating all the rules, if UM is unable to find a rule whose conditions are met, UM will offer the caller to leave a voice message.
Figure 6: How UM call answering rules are evaluated

Call answering rules are an Exchange 2010 feature, and only available to UM-enabled users with a mailbox on an Exchange 2010 Mailbox server.

-Chun Yong Chua

Fixed: Message size issue in Outlook 2010 Beta

2010-02-12T18:54:00Z

The Outlook team recently reported an issue on their blog about message sizes in the Microsoft Outlook 2010 Beta. A bug in the Microsoft Office 2010 Beta increases the message size of some e-mail messages sent from Outlook 2010 Beta.

The Office team has been hard at work building a fix for this issue, and it's now available on the Outlook blog. Note, the fix does not resize existing messages. To download the fix, and get some tips on how to reduce the mailbox size, head over to Fix for issue with message sizes in Outlook 2010 Beta on the Outlook team blog.

-Bharat Suneja

How to recreate the legacy Exchange Virtual Directories in Exchange 2010?

2010-02-10T22:50:34Z

The Exchange 2010 install of the CAS role includes the installation of the legacy Exchange Virtual Directories. The Exchange, Public and Exchweb Virtual Directories are placed and configured directly in the IIS metabase, therefore these Virtual Directories are only local to the Server and no longer objects located in Active Directory.

Additionally, the Exchange 2010 New-OWAVirtualDirectory cmdlet no longer recognizes the -Name parameter and is hard coded to use "OWA" for the name. This behavior is by design.

NOTE: This behavior is different than that in Exchange 2007 where it was necessary to use the New-OWAVirtualDirectory in order to recreate the Exchange, Exchweb, and Public Virtual Directories.
NOTE: Running the cmdlet Remove-OWAVirtualDirectory will remove the Exchange, Exchweb, and Public Virtual Directories in addition to the OWA Virtual Directory. The New-OWAVirtualDirectory cmdlet will create a new OWA Virtual Directory as well as the Exchange, Exchweb, and Public Virtual Directories.

So what happens if we need to manage or recreate only the legacy Exchange Virtual Directories on an Exchange 2010 Server?

Since the legacy Exchange Virtual Directories are no longer listed and manageable in Active Directory, we need to use the appcmd utility (as we do to manage other Virtual Directories in IIS 7 and above) or add the Virtual Directories in IIS manually and modify the applicationHost.config file directly.

Option 1:

In order to recreate the legacy Exchange Virtual Directories using appcmd, take the following actions for each Virtual Directory (Exchange, Exchweb, and Public):

1) Open a command prompt, change directory to %SystemRoot%\System32\inetsrv

2) Replacing Default Web Site with the appropriate Site Name (if different) and <VDir> to the correct legacy Exchange Virtual Directory alias, Run the following commands:

appcmd delete vdir /vdir.name:"Default Web Site/<VDir>" /commit:APPHOST
appcmd add vdir /app.name:"Default Web Site"/ /path:/<VDir> /physicalPath:"<Exchange Install Directory>\V14\ClientAccess\Owa" /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:directoryBrowse /enabled:false /showFlags:"Date, Size, Extension, LongDate" /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:handlers /accessPolicy:Read /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:windowsAuthentication /enabled:false /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:anonymousAuthentication /enabled:true /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:digestAuthentication /enabled:false /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:basicAuthentication /enabled:false /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:defaultDocument /enabled:true /commit:APPHOST
appcmd set config "Default Web Site/<VDir>" /section:httpRedirect /enabled:true /destination:/owa /exactDestination:false /childOnly:false /httpResponseStatus:Found /commit:APPHOST

NOTE: I recommend putting the above into a batch file and make 3 copies. Modify each batch to replace the <VDir> with Exchange, Exchweb, or Public.

Option 2:

In order to recreate the legacy Exchange Virtual Directories manually, take the following actions for each Virtual Directory (Exchange, Exchweb, and Public):

1) Open IIS Manager, expand Default Web Site, right click on the <VDir> and choose Remove

2) Highlight the Default Web Site, and choose View Virtual Directories in the Actions pane.

3) In the Actions Pane, choose Add Virtual Directory... Add the correct Alias (Exchange, Exchweb, or Public) and Physical Path which is the "<Exchange Install Directory>\V14\ClientAcess\Owa" and click OK.

NOTE: Default Web Site is set to Anonymous by default, therefore the newly created Virtual Directories should also be set to Anonymous. If not, then modifying the applicationHost.config below will correct the authentication.

4) Once the above is complete, open the applicationHost.config file using notepad which is located in %SystemRoot%\System32\Inetsrv\config\

5) Search for the following: "Default Web Site/owa/14". Once found, look for the </location>, After the </location>, copy and paste the following text renaming <VDir> to the legacy Exchange Virtual Directory alias:

After recreating the legacy Exchange Virtual Directories and modifying the applicationHost.config, open a command prompt and run:

Iisreset /noforce

- Steve Swift

What's new with Exchange 2010 languages support?

2010-02-09T16:30:00Z

With the launch of Exchange 2010, we have introduced a new way to manage the set of languages available for Exchange.

In previous versions of Exchange, you had one server language per DVD. In Exchange 2010, we have changed to a new language neutral design. This allows us to release all available languages in a single DVD. We can also add and update additional languages over the lifetime of the product. This model also allows us to easily update the translation quality over the lifetime of the product. We are no longer dependent on component updates or service pack deployments.

We provide this flexibility through the use of a language pack bundle. The Exchange 2010 RTM DVD contains our initial version of this bundle. Exchange Setup will install this bundle from the DVD, from a network share, or recommend you to download the most up to date version of the bundle if your server has internet access. Since Exchange 2010 RTM, we have released an updated build of the bundle, available from here. Our recommendation is that all customers deploy this updated bundle as it contains both additional client languages, as well as translation improvements for all the languages. An up to date list of supported languages can be found in Exchange 2010 Language Support. Going forward, we do expect to release updated language pack bundles to improve quality and add additional languages.

The language bundle is delivered as a single downloadable file which includes all localized resources for the server— including Exchange Management Console (EMC) and client (OWA & ECP) interfaces within Exchange. To get a localized experience in EMC, ECP and OWA, you need to install the language pack. Listed below are three scenarios that we would like to call out and provide guidance around them with regards to language pack installation.

Language pack bundle deployment as part of the initial server setup
This scenario applies to an environment where you are installing the language during the fresh setup of your server. We recommend you to install the Language pack in this scenario from the DVD. While the installation of the bundle is available from the usual graphical setup, commands line options are also available. For this scenario use:
Setup.com /mode:install /roles:.... [/LanguagePack:]
Language pack bundle deployment for the first time post initial server setup
This scenario applies to you, if you have set up your server first and now want to install the Language pack , we recommend you to download the latest Language pack from the download center and follow the steps therein. If you prefer to use the command line option, use :
Setup.com /mode:install /LanguagePack:
Language pack update
You are already using the language pack bundle, and now want to update it to the latest available. In this scenario, update the exiting bundle with the latest Language pack from the download center and follow the steps therein. In addition, you could also install it using the command line:
Setup.com /mode:upgrade [/LanguagePack:]

Note:
- Before installing the language pack bundle, ensure that your OS is installed in the language that you would like to use Exchange in.
- You should deploy the new or updated language pack bundle starting with your mailbox servers.
- After installing the Exchange language packs, restart the computer to complete the installation of the language packs.
- If there is no language pack bundle deployed, English will be the only language available for Client and Server, regardless of the OS Language.

-Kevin Allison

Troubleshooting Exchange 2010 Management Tools startup issues

2010-02-04T19:58:41Z

In this blog post, we will be highlighting some of the most common errors that may be seen when attempting to open the Exchange Management tools (Exchange Management Console and Exchange Management Shell).

To start off, you first need to be aware that in Exchange 2010, all management is done via Remote PowerShell, even when opening the Management Tools on an Exchange server. Where this differs from Exchange 2007 is that there is now a much larger dependency on IIS, as Remote PowerShell requests are sent via the HTTP protocol and use IIS as the mechanism for connections. IIS works with the WinRM (Windows Remote Management) service, and the WSMan (Web Services for Management) protocol to initiate the connection.

When you click on the Exchange Management Shell shortcut, a Remote PowerShell session is opened. Instead of simply loading the Exchange snap-in (as we did with Exchange 2007), PowerShell connects using IIS to the closest Exchange 2010 server via WinRM. WinRM then performs authentication checks, creates the remote session and presents to you the cmdlets that you have access to via RBAC (Role Based Access Control).

Since all Remote PowerShell connections go through IIS, we have identified some of the most common errors that may be exhibited when attempting to open the Exchange Management tools along with the most common causes of those errors and how to address these issues. We have attempted to list these in order of frequency.

Issue:

Connecting to remote server failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.

Possible causes:

1. Remote PowerShell uses Kerberos to authenticate the user connecting. IIS implements this Kerberos authentication method via a native module. In IIS Manager, if you go to the PowerShell Virtual Directory and then look at the Modules, you should see Kerbauth listed as a Native Module, with the dll location pointing to C:\Program Files\Microsoft\Exchange Server\v14\Bin\kerbauth.dll. If the Kerbauth module shows up as a Managed module instead of Native, or if the Kerbauth module has been loaded on the Default Web Site level (instead of, or in addition to, the PowerShell virtual directory), you can experience this issue. To correct this, make sure that the Kerbauth module is not enabled on the Default Web Site, but is only enabled on the PowerShell virtual directory. The entry type of "Local" indicates that the Kerbauth module was enabled directly on this level, and not inherited from a parent.

2. If the WSMan module entry is missing from the global modules section of the C:\Windows\System32\Inetsrv\config\ApplicationHost.config file, as follows:

<globalModules>
<add name="WSMan" image="C:\Windows\system32\wsmsvc.dll" />

This will result in the WSMan module displaying as a Managed module on the PowerShell virtual directory.

To correct this, make sure that the WSMan module has been registered (but not enabled) at the Server level, and has been enabled on the PowerShell virtual directory.

3. If the user that is attempting to connect is not Remote PowerShell enabled. To check if a user is enabled for Remote PowerShell, you need to open the Exchange Management Shell with an account that has been enabled, and run the following query.

(Get-User <username>).RemotePowershellEnabled

This will return a True or False. If the output shows False, the user is not enabled for Remote PowerShell. To enable the user, run the following command.

Set-User <username> -RemotePowerShellEnabled $True

Issue:

Connecting to the remote server failed with the following error message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more information, see the about_Remote_Troubleshooting Help topic.

Possible Causes:

1. The http binding has been removed from the Default Web Site. A common scenario for this is if you are running multiple web sites, and attempting to set up a redirect to https://mail.company.com/owa by requiring SSL on the Default Web Site, and creating another web site to do the redirect back to the SSL-enabled website.

Remote PowerShell requires port 80 to be available on the Default Web Site. If you want to set up an automatic redirect to /owa and redirect http requests to https, you should follow the instructions located at http://technet.microsoft.com/en-us/library/aa998359(EXCHG.80).aspx and follow the directions under the section "For a Configuration in Which SSL is Required on the Default Web Site or on the OWA Virtual Directory in IIS 7.0".

2. The http binding on the Default Web Site has been modified, and the Hostname field configured. To correct this issue, you need to clear out the Hostname field under the port 80 bindings on the Default Web Site.

Issue:

Connecting to remote server failed with the following error message: The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic. It was running the command 'Discover-ExchangeServer -UseWIA $true -SuppressError $true'.

In addition, you may see the following warning event in the System log:

Source: Microsoft-Windows-WinRM
EventID: 10113
Level: Warning
Description: Request processing failed because the WinRM service cannot load data or event source: DLL="%ExchangeInstallPath%Bin\Microsoft.Exchange.AuthorizationPlugin.dll"

Possible Causes

1. The ExchangeInstallPath variable may be missing. To check this, go to the System Properties, Environment variables, and look under the System variables. You should see a variable of ExchangeInstallPath with a value pointing to C:\Program Files\Microsoft\Exchange Server\V14\.

2. The Path of the Powershell virtual directory has been modified. The PowerShell virtual directory must point to the \Program Files\Microsoft\Exchange Server\v14\ClientAccess\PowerShell directory or you will encounter problems.

Issue:

Connecting to remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests on the correct port and HTTP URL. For more information, see the about_Remote_Troubleshooting Help topic.

Possible Causes:

1. Make sure the MSExchangePowerShellAppPool is running. If it is, try recycling the Application Pool and check for errors or warnings in the Event logs.

2. Make sure that the user that is trying to connect is Remote PowerShell Enabled (see the first error for details on how to check this).

3. Make sure WinRM is properly configured on the server.

a. Run WinRM Quick Config on the server and ensure that both tests pass and no actions are required. If any actions are required, answer Yes to the prompt to allow the WinRM configuration changes to be made.

b. Run WinRM enumerate winrm/config/listener and ensure that a listener is present for the HTTP protocol on port 5985 listening on all addresses.

Issue:

Connecting to remote server failed with the following error message: The WinRM client received an HTTP status code of 403 from the remote WS-Management service.

Possible Causes:

1. The "Require SSL" option has been enabled on the PowerShell Virtual Directory. To resolve this, remove the "Require SSL" option from this Virtual Directory. The Exchange Management Tools connect over port 80, not 443, so if Require SSL is set, when a connection is attempted on port 80, IIS will return a 403 error indicating SSL is required.

- Ben Winzenz, Solange Trombini

Triaging Exchange Performance issues related to Active Directory (AD) Performance (including Networking and DNS performance)

2010-02-03T18:03:13Z

In many organizations the skill sets that administer Exchange, Active Directory (AD), Domain Naming System (DNS), and the network infrastructure are often segregated. Since Exchange is heavily dependent on communication with AD and that communication must traverse the network, the ability for the Exchange Administrator to have a basic understanding of how each of these components can be troubleshot is useful. Furthermore, often times the DNS, network, and AD administrators are not familiar with how Exchange is dependent on their services and thus are not fully prepared to help an Exchange administrator isolate production issues. As such, by the end of this it is hoped that there will be enough information to allow the Exchange, DNS, network, and AD administrators to each collect the data necessary to identify the majority of the issues.

There is some required reading that is necessary in order to understand much of the content that follows. Rather than reiterate all of the pre-requisite knowledge here, reviewing the following content might be helpful:

To start with, an understanding of what the performance counters "LDAP Read Time" and "LDAP Search Time" track is very useful (as identified in Ruling Out Active Directory-Bound Problems and Monitoring Common Counters: Exchange 2007 Help). One knows that once these counters approach or exceed the recommended thresholds, troubleshooting outside of Exchange needs to begin. Therefore, the question, "What are the potential contributors to high values for these counters?" is thus raised. Since these counters are installed with Exchange and are tracked within the DSAccess /ADAccess components, depending on the version of Exchange, there are a number of dependencies that Exchange does not have visibility into that the troubleshooter needs to be cognizant of when evaluating where the source of the delay is incurred. In order to provide a clearer picture of what needs to be considered, below is a walkthrough of a somewhat simplified list of events that happens after the clock starts ticking (reference How Active Directory Searches Work for greater detail):

1. WLDAP32.DLL receives the request from one of the Exchange processes. It has to locate a Global Catalog (GC) first.
2. DNS query traverses the network. Unresponsiveness or degraded responsiveness from DNS servers will degrade the overall performance of the query.
3. WLDAP32.DLL submits the query to the GC.
4. If not already started, a Transmission Control Protocol (TCP) session is established, the Lightweight Directory Access Protocol (LDAP) query traverses the wire.
Note: since TCP requires a 3 stage handshake before the session and in turn windowing can work, multiply network latencies by 3 to figure out how much time it takes just to establish the session (thus that 10 ms latency becomes 30 ms delay even before the Exchange server submits the query).
5. The TCP data transmitted up the networking stack to LSASS.exe listening on the LDAP ports.
6. The query is processed by the GC and searches through the database to return the results.
7. The data is sent out the Network Interface Card (NIC) on the GC.
8. The query is received via WLDAP32.DLL from the GC.
a. If there are multiple pages all pages need to be returned and stored in a data structure within DSAccess / ADAccess. Each of these requires traversal of the network and resubmission of the query.
b. If there are a large quantity of values in the requested attribute (think group memberships), these all need to be retrieved and returned. This results in multiple queries to the GC. This may require steps 1 - 5 to recur.

As can be seen from the above set of steps a lot of the performance delays are heavily dependent on network latency and network performance. In fact the only portion of that number that is actually impacted by the GC performance itself is step 5. As such, in addition to troubleshooting any potential performance issues on the GC, it is highly advantageous to collect network trace data when issues do occur, preferably from both sides of the conversation.

Summary of Data needed to troubleshoot

Exchange Server - Having this data from both sides is important for comparison purposes (i.e. is the processor spike on the DC correlating with the degraded performance in Exchange)

Performance Counters specified in the referenced articles:
- Exchange 2007/2010 - Monitoring Common Counters: Exchange 2007 Help
- Exchange 2000/Exchange 2003 - Ruling Out Active Directory-Bound Problems
  Note: For Exchange 2003 add in MSExchangeDSAccess Domain Controllers\LDAP Search Time(All Instances) and MSExchangeDSAccess Domain Controllers\LDAP Read Time\(All Instances) so that the DC to be troubleshot can be identified.
Network Trace from the Exchange server side

Note: To ease collection of the Exchange performance data reference "Mike Lagase : Perfwiz replacement for Exchange 2007"

AD Server -

Performance Data -
- Windows 2008/2008 R2 - Monitoring Common Counters: Exchange 2007 Help recommends starting the built in Data Collector Set "Reliability and Performance\Reports\System\Active Directory Diagnostics" and use the data collector set.
  Reference: Performance Logs and Alerts
- Windows 2003
  - At a minimum, the counters specified in Ruling Out Active Directory-Bound Problems
  - SPA - Select the Active Directory report.
Network Trace from the affected DC side.

DNS Server - not all organizations use Microsoft DNS servers, but for the purposes of this article the assumption that a Microsoft DNS server is in use.

Network Trace from the DNS server
Performance Data
- Windows 2008/2008 R2 - start the built in Data Collector Set "Reliability and Performance\Reports\System\System Performance" and use the data collector set.
- Windows 2003 - SPA Select the System Overview report
- Performance counters - All DNS related counters.

How to triage where troubleshooting needs to happen

Network Infrastructure issues -
- Use the network trace to determine if the DNS server is responding in a timely fashion. Keeping in mind that Microsoft recommends the entire conversation happen in less than 50 ms in general, if the DNS servers are taking more of that 50 ms than the actual LDAP search, this could be problematic. Timely is also subjective to whatever is normal for the environment and base lining is helpful here. In general if the DNS server is on the same segment and performing properly, 1 to 2 ms or less is reasonable. Of course as always, it is recommended that the environment be base lined so that the expected performance is known.
- Use the network trace to determine if the TCP session with the GC is being established in a timely fashion.
  Again, keeping in mind that there is a 50 ms performance threshold, if the TCP session establishment is taking more time than the actual LDAP search takes to complete, this could be problematic. Also, timely is also subjective to whatever is normal for the environment and base lining is helpful hear. In general, if the GCs are on the same segments and both the network and the GC are performing properly, sessions established in 1 to 2 ms or less is reasonable. Of course as always, it is recommended that the environment be base lined so that the expected performance is known.
AD Issues - Often AD infrastructures are large and the AD admins cannot determine which of the many GCs is in use by the affected Exchange server, thus the responsibility of pointing the AD support team to the right box to troubleshoot falls on the Exchange admin.
- Use the network trace to find out which GC is having a large delay between the LDAP Request and LDAP Response packets.
- Use the performance counters to narrow down to the box that is performing poorly.
  - Exchange 2007/2010 - MSExchange ADAccess Domain Controllers\LDAP Search Time and MSExchange ADAccess Domain Controllers\LDAP Read Time check the specific instances for values exceeding the recommended thresholds. This will report the specific GC (and underlying network path) that is not responding to Exchange as quickly as desired.
    Note: On Exchange 2007 SP2 and newer systems, the "MSExchange ADAccess Local Site Domain Controllers" can also be used in the same manner.
  - Exchange 2003 - MSExchangeDSAccess Domain Controllers\LDAP Search Time and MSExchangeDSAccess Domain Controllers\LDAP Read Time, check the specific instances for values exceeding the recommended thresholds. This will report the specific GC (and underlying network path) that is not responding to Exchange as quickly as desired.
  - Exchange 2000 - performance counters similar to the above did not exist. The only way to determine which GC is responding slowly is to analyze a network trace as described previously.

Next, identify bottlenecks within the GC (note: for the most part these basics apply to troubleshooting DNS performance as well):

While there are a large variety of potential issues, there are some basic things that can be eliminated before advanced troubleshooting need take place. The below guidance are the most basic remediation suggestions for the largest variety of these types of issues. Essentially this comes down to, "Ok, so the issue is identified, now what do I do?"

Since there is plenty of published information on how to isolate disk, processor, and/or network issues, please reference the excellent articles linked at the beginning for the recommended thresholds. Also, check out Performance Analysis of Logs (PAL), which automates the analysis of the performance data collected to help determine where the system is bottlenecking.

Disk bound - Once the issue is isolated to being disk bound, there are essentially two options, add memory or allow for increased disk IO (i.e. add spindles for direct attached storage).

Adding memory is not quite as cut and dry as just throwing in some sticks of RAM.
- 32-bit systems - It makes sense to avoid rehashing the entire discussion of 32-bit OS architecture especially since 32-bit architecture is on its way out. Reference this support article for some useful info. Memory usage by the Lsass.exe process on domain controllers that are running Windows Server 2003 or Windows 2000 Server
- 64-bit systems - Generally the easiest thing to do is to put enough RAM in the box to load the entire database (minus white space) in memory. As of this writing, even for the largest enterprises (16 GB to 32 GB databases), this is becoming quite inexpensive or easy to justify. If the box has enough memory to load the entire database already and disk IO is the issue, troubleshoot why the database is not being loaded. A couple tip:
  - Measure Process\Virtual Bytes\lsass to see if it is about (within 10%/20%) of the size of the database (minus white space) or larger. If it isn't, troubleshoot why lsass isn't growing.
    Ensure that other applications aren't consuming large amounts of memory and preventing LSASS from growing like it wants to.
Increasing disk-throughput.
- If RAM cannot be added (or additional RAM will provide no benefit) due to the limits of the 32-bit OS architecture migrate to 64-bit. If migrating to 64-bit is not an option, the only option left is to add more disk throughput. Assuming that the storage is local and not SAN attached, it essentially means add more spindles (talk to the SAN guys if the AD database is stored on the SAN).
- Reducing disk IO to the database volume is another option. Look at options such as turning off backups and AV scans during production hours, move SYSVOL and/or the database logs to separate spindles. Just follow standard storage performance tuning recommendations. Also, IOs to the disk can be reduced by reducing the demand on the boxes either by adding more DCs to the environment or reducing client demand. SPA and Performance Logs and Alerts will help track down what else is using the volume.

Processor bound:

Ensure that LSASS is the process using the CPU, if not eliminate the problematic process.
If LSASS is running hot, then use SPA (Windows 2003) or Performance Logs and Alerts (Windows 2008 and up) to identify where the CPU is being consumed within LSASS.
If everything looks normal then the only option remaining is to add more processors to the box or more boxes to the infrastructure if there is high processor load across multiple systems.
Note: In large infrastructures with multiple DCs and multiple Exchange servers, adding more hardware only makes sense if all of the boxes are experiencing high CPU utilization (close to or exceeding the thresholds in Ruling Out Active Directory-Bound Problems). If most or all of the boxes in the Exchange site are not experiencing high CPU utilization then there is probably a system specific reason for high CPU time. Look for hardcoded applications, improperly balanced load due to tuning LDAP Weights and Priorities (Check out Why you need Active Directory for Exchange Server 2007 for details on how this works), or anything else which may indicate why on GC is being singled out compared to the rest. Fix that issue before adding more hardware.

Network:

If the outbound queue length is > 0 sustained the easy fixes are:
- Update the NIC drivers to the latest versions if possible
- Disable the Scalable Networking pack features (SNP) on Windows 2003 base servers- An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers. Check the network card settings to see if your driver can enable or disable certain SNP offloading features. Depending on the driver, if you disable these settings at the OS level, there is a possibility that they can still be enabled directly on the network card settings.
- If the DCs are running Windows 2003 and the Exchange Servers are running on Windows 2008, then apply the Scalable Networking Pack rollup on the DCs to ensure that known compatibility issues do not occur. Also be aware of a known Windows 2008 TCP/IP setting issue in http://support.microsoft.com/kb/967224 that can affect overall network connectivity and performance.
- Ensure that the NIC speed/duplex settings are properly set. MS recommendations, as of this writing, recommend both the GC and switch be set to 100/Full for 100 Mb segments and Auto/Auto for GigE segments. Check the Network Interface\Current Bandwidth performance counter to ensure that network card auto-negotiation is not changing the speeds the network card. This value should remain consistent at one value throughout the lifetime of the server being online.
- Disable the load balancing/fault tolerance features of the NIC driver suite.
Ensure that Network Interface\Packets Outbound Errors performance counters remains at 0 all the time. Any increases in this value can indicate intermittent connectivity issues on the server.
If the network connection is using more 60% to 80% of the maximum bandwidth, the only solutions are to upgrade the available bandwidth (100Mb > GigE) or distribute the load across multiple servers/network connections.

Identifying the source of the GC load

The possibility exists that, despite the DC being either disk or processor bound, that the load is unnecessary. To determine this, the load on the box must be analyzed in greater detail than just the performance can provide. To that end, there are two main strategies for identifying the load on the box, both of which can be used concurrently. How to analyze this data is somewhat out of scope of this article, but the below bullets contain links to useful articles on how to use this data:

Enable diagnostic logging on the DC by setting the "15 Field Engineering" registry value in order to identify expensive queries that return greater than 1,000 objects.
- For more details on how to use this reference:
  - Logging LDAP searches: AD and ADAM > ActiveDir.org
  - Creating More Efficient Microsoft Active Directory-Enabled Applications
- Pros
  - Shows the expensive or inefficient LDAP queries
  - The data is easy to get to since it is stored in the DS log
  - Data collection and alerting can be automated using SCOM.
- Cons
  - Can overwrite the logs quickly, but the irony here is that if it is flushing the logs quickly, there is probably a lot of expensive queries that need to be remediated.
  - Doesn't actually track the processor impact of the query.
  - Only reports on LDAP queries.
Use SPA (note this functionality exists natively in Windows 2008 and is entitled Performance Logs and Alerts which is part of Windows Reliability and Performance Monitor)
- More details
- Pros
  - Shows everything that is going on, whether it is authentication, LDAP queries, network traffic, or disk IO.
  - Data collection can be triggered using SCOM when certain thresholds are hit.
- Cons
  - On busy DCs the logging can quickly grow to unmanageable levels (busy being hundreds of LDAP queries and authentications per second). Think on the order of 100+ MB per minute. But this isn't too bad as usually a 5 to 10 minute data collection during a production issue will get the data necessary to identify what is occurring.
  - Requires the installation of a tool set (Windows 2003 only)

Conclusion

Data collection is key! Getting the right data when the problem occurs is absolutely critical to the restoring the health of the environment. Furthermore, as Exchange is the product suffering when its dependencies are not responding adequately, it falls on the Exchange administrator to do the leg work collecting the right data from the perspective of Exchange, and then isolating where the network, DNS, and AD administrators might need to look and start collecting data. Without this information, the network, DNS, and AD admins will be looking for a needle in a haystack and it is highly unlikely that they will find anything of use. Thus, once the Exchange admin has isolated the dependencies of concern, using the information provided here, it will be much easier to collect the data necessary from those systems to remediate any issues.

Thanks to Rod Fournier and Mike Lagase for their endless suggestions for improvement of this blog post.

- Ken Brumfield

You Had Me At EHLO...

Exchange Scalability Limits Worksheet Available

GAL Photos in Exchange 2010 and Outlook 2010

A minor schema change

Loading pictures into Active Directory

Thumbnails in Outlook 2010

GAL Photos and the Offline Address Book

Dude, where's my rollup?

Released: Update Rollup 2 for Exchange Server 2010

Only the English Rollup?

Known Issue

Removing a Feature From The Exchange Control Panel

Remove the Delivery Reports tab for a user

Exchange Server in Depth: Master Concepts and Optimize Your Server Environment

Announcing the Release of the Exchange Server Pre-Deployment Analyzer

Dude, Where's My Single Instance?

E2010 Mailbox Server Role Requirements Calculator v4.5

Call Answering Rules (Part 2): Call Answering Conditions

The Call Answering Rule form

Different classes of conditions

Caller Identity

Time of Day

Calendar Free/Busy status

When my automated email reply is turned on

Call Answering Rules and conditions

Multiple Call Answering Rules

Fixed: Message size issue in Outlook 2010 Beta

How to recreate the legacy Exchange Virtual Directories in Exchange 2010?

What's new with Exchange 2010 languages support?

Troubleshooting Exchange 2010 Management Tools startup issues

Connecting to remote server failed with the following error message: The WinRM client received an HTTP status code of 403 from the remote WS-Management service.

Triaging Exchange Performance issues related to Active Directory (AD) Performance (including Networking and DNS performance)