You Had Me At EHLO...

Exchange Server 2007 Server Installation Templates and Build Automation Guidance Released

Exchange — Thu, 07 Aug 2008 19:46:00 GMT

We have always recommended that customers prepare formal server build documentation for their Exchange environments. Until now, we've not provided any formal guidance around what that documentation should look like. We now have build documentation templates and instructions for preparing a build automation DVD. You can use these templates as a starting point for formally documenting your Exchange server builds. Preparing a build automation DVD can help streamline the installation of Exchange 2007 on both Windows Server 2003 and Windows Server 2008. The templates and build automation guidance can be found here: http://technet.microsoft.com/en-us/library/cc533547(EXCHG.80).aspx. Thanks to Ross Smith IV, with Exchange Center of Excellence, for his extensive help with this release.

Thanks,
Tom Di Nardo

Share this post :

Understanding Exchange 2007 Memory Usage and its use of the Paging File

Exchange — Wed, 06 Aug 2008 20:48:00 GMT

Lately, we've been getting a lot of questions related to Exchange 2007 memory consumption and why Exchange uses so much of the paging file. Well, there are some valid reasons why and some misconceptions that need to be corrected.

Comparing memory usage between Exchange 2007 and Exchange 2003
First, let's start with why Store.exe uses so much RAM. If we take a step back in time to the Exchange 2003 era, this blog was quite active on how to tune memory on an Exchange 2003 server. I am not going to go in to specifics or great detail, but one thing to call out is that on the 32-bit architecture, we were limited to addressing 4GB of virtual memory on any given server. So essentially, any 32bit program could address up to 4GB of virtual memory. This address space is typically split so that applications could address 2GB of memory and 2GB would be for the Windows Executive. By adding the /3GB switch in the boot.ini, applications could now access up to a 3GB virtual address space and lower the Executive down to addressing only 1GB, essentially halving the memory that is can be addressed or is available for kernel drivers, paged/non-paged pool memory, PTE's etc. The larger the load that you put on a server has the potential to exhaust important resources on the server which eventually causes a failure or server outages. Any type of memory leak in these areas could be detrimental to the stability of the server. With the numerous posts on this blog regarding the /3GB, /USERVA and /PAE switches, what drivers to update, how to tweak memory to allow Exchange to run optimally, etc., it was always a juggling act to try to keep Exchange servers online. Cluster servers were especially touchy as http.sys will start rejecting HTTP connections when non-paged pool hit a certain level, thus causing cluster failovers.

Due to these constraints of the 32-bit architecture, we needed a way to balance I/O based applications on the server. Essentially, there are two types of I/O for the Windows operating system, Buffered and Unbuffered I/O. For Buffered I/O, only portions of the working set can be contained in memory; those portions not in use are retained in the page file. For Unbuffered I/O, the entire working set can be contained in memory. On any given system, we needed to maintain a balance so that each would get the right resources that they need. This is one of the main reasons why we needed to put a maximum on the ESE database cache to leave us enough room for the system cache.

In most implementations, the default of 900MB of database cache was used which was sufficient for most deployments, but some implementations had the need for more cache increasing it to a maximum of 1.2GB of memory on systems of 2GB or more of physical RAM. This maintained the balance that was necessary so that the system cache could perform efficiently.

So you may ask, why did I go through all of the above to explain how Exchange 2003 worked? Well, one of the aspects of Exchange 2007 that people ask about is the difference between Exchange 2007 memory usage to Exchange 2003. Indeed, the memory usage is much higher,and there are very good reasons for this. Let me explain further.

Exchange 2007 Design Considerations
During the design phase of Exchange 2007, we needed to find a way to scale out servers further than Exchange 2003 did due to the rising demands of email usage in companies. For many companies, email is their life line and if an exchange server should go down, they could be losing money at an astronomical rate. One of the performance bottlenecks in Exchange 2003 was disk latencies which was mostly due to the memory constraints of the 32-bit platform. Ultimately, -, we had to read more from disk than what was available in memory. This required a well performing disk subsystem to handle the sustained I/O necessary to respond to client requests in a timely manner. In many cases where disks were the underlying problem, a re-design of the disk subsystem was necessary to allow the current amount of IOPS that were being generated on the server to be handled by the disks. This could range from adding more spindles to the disk arrays, rebuilding different RAID solutions to provide enough sustained throughput based on current user load.

Why we learned to love the 64-bit Platform
With the 64-bit architecture being available, we made a conscious decision that Exchange 2007 would only be supported in production on 64-bit hardware. This was mainly done to overcome the limitations that the 32bit platform had. The design goal here was to reduce cost of Exchange deployments in which storage turned out to be the number one driver of cost and support overhead. By moving to the 64-bit platform, we are able to reduce our I/O requirements thus bringing us back to easier balancing of I/O and capacity requirements.

Now, one of the results of being able to address more memory is the capability to cache more memory for each application. To allow this to happen for Exchange, the ESE Database Cache Size limit was removed to allow Exchange the ability cache more pages in memory. Accessing pages in memory is extremely fast, so the more data we cache in memory, the less we actually have to read andwrite from the disk. When following our best practice guidance around storage group deployment and memory sizing, Exchange 2007 reduces the amount of I/O required overall. This gave us rather huge performance gains. For more information regarding I/O improvements on the 64-bit platform for Exchange, see the following blog post http://msexchangeteam.com/archive/2006/09/08/428860.aspx.
With no limitation in Database Cache for Exchange 2007, the memory usage for the store process will naturally be much higher compared to what Exchange 2003 used due to the many benefits discussed earlier. Memory allocation for the ESE cache is dynamic, so Jet will use all available memory on a system and return it as needed. For example, if a server had, let's say, 16GB of physical RAM installed, the database cache could consume approximately 14GB of memory, roughly 2GB less than the total amount of physical RAM in the server, leaving enough memory for the system cache or other applications running on the server.

The algorithm is relatively simple for the ESE Database Cache. The DBA (Dynamic Buffer Allocation) will grow the cache by comparing the amount of I/O the databases are doing with the amount of I/O the system is doing in terms of hard page faults (Memory\Transition pages repurposed/sec). So if the database is doing more I/O, and Exchange is hitting hard page faults, we will increase the cache. If the database is doing less I/O than the hard page faults, then we will decrease the cache. The goal is to keep the ESE cache in balance with the disk cache so we don't end up paging ourselves to death. In some cases, on larger servers, Online maintenance during the night could cause up to a 4GB swing in memory for the Database cache alone due to the aggressive nature of how we touch pages during this process. (It may also be prudent to stagger your OLM schedules to assist with this.) Each server configuration is different, so it is important to have a monitoring solution in place or a Performance monitor log to track memory usage over time if memory pressure is occurring. If memory usage gets to a point where some driver or application attempt to allocate a large block of contiguous memory, and there is not one available, then the OS may decide to start trimming working sets which is a really bad thing performance wise. I've discussed this on my own blog http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx and possible remediation's to work around those type scenarios if you fall in to this bucket.

Other Server Memory Considerations
So we've talked a lot about memory usage for ESE database cache, but this cache alone is not the only significant consumer of memory on an Exchange server, it is just the majority. From an Exchange perspective, we have other processes going on such as Content Indexing, log shipping in continuous replication and underlying .NET applications that Exchange makes use of now.

Most of Exchange has been rewritten in managed code using the .NET framework except for Store, DAV, and DSAccess. Managed code offers some significant advantages over non-managed code such as the ability to compile applications in real-time so that you don't have to worry about running your application on different architectures or platforms and the ability to manage memory efficiently. .NET applications use the Common Language Runtime (CLR) to allow easier coding in different languages because they share the same runtime. Any code that you develop with a language compiler that targets the runtime is called managed code as mentioned here: http://msdn2.microsoft.com/en-us/library/ddk909ch.aspx.

Reasons for Memory Paging
One of the primary reasons for memory paging in Exchange 2007 is Content Searching, which uses the system cache since this is buffered. Performing more searches increases the file cache you have which in turn reduces the amount of IOPS that are generated. Content Indexing itself does not make use of the file cache as this is designed to work on a memory structure that is not affected by the file cache. Master merges of index data does use a small amount of the file cache, but it very short lived. Database I/O is actually unbuffered, so this not affected by the system cache. This is why the ESE cache doesn't continuously grow (ESE cache I/O is less than the system cache I/O). It's being balanced out by the content searching I/O (and potentially other applications that use buffered memory I/O) and thus the system cache. CCR log shipping is also unbuffered while log files are being copied over SMB, so this is also not affected by the system cache.

Now couple all of this memory management with other 3rd party applications and drivers that are installed on the server which are all competing for the same resources, you can see that this is vastly different than what Exchange 2003 used to be. There is actually no comparison at all.
How Exchange makes use of the Paging File

So this brings up the second part of this document where I will now talk about the paging file. So you may ask, if I have so much memory in the server, why is a paging file needed at all? Our system requirements talk about needing one for getting a memory dump on the server, but what does Exchange really use it for? I get this very question asked very frequently. In certain cases, it is true that Windows 2003 does not require a paging file. It is however completely incorrect that Exchange does not need a paging file for normal runtime operations when running on top of that OS. There is always going to be a need to page out unused or rarely used memory to the page file so that we can make maximum use of RAM. For example, there could be a large amount of VM pages used during system startup that are not useful under main load. Unexpected transient conditions could cause large upward swings in memory that could cause us to spill pages of memory in to the paging file. There are enough processes outside of the cache manager that consume a significant amount of memory in Exchange 2007 (e.g. ESE and all processes that utilize the CLR) that there is a need for a substantially sized paging file as the system balances the memory footprint of the competing processes. Without a paging file (or with one that is too small) there is a very high chance that applications will encounter OOM (out of memory) conditions and both the performance and the functionality of the server will suffer and may lead to a possible blue screen.

The cost of having a 32GB paging file on a server with 32GB of physical RAM on disk is pretty minimal, so having a properly configured paging file (RAM+10MB) will be able to sustain any unexpected growths in memory where we need more commit charge to handle current load. The other benefit is that if we should get ourselves in to a working set trimming issue, we would at least have the ability to flush these pages out to the paging file and then page them back in when a particular page needs to be accessed again. Excessive working set trimming on an Exchange 2007 server is not normal and should be investigated. Again, check out my blog post on working set trimming problems and why this is bad for performance.

Why does Exchange allocate so much memory in the paging file?
Another question that generally comes up is why does Exchange allocate so much memory in the paging file? Depending on the amount of memory that is consumed for the store process, we will need to at least allocate enough memory in the paging file to match the current working set that is in RAM on the server. If a system wide working set trim should occur, we would be able to page the current pages in the store working set out to the backing store or paging file. It is important to monitor Memory\Pages/sec on the server to detect if this working set trimming problem is causing any performance related problems on the server. You should be able to see by adding Process\Working Sets a sharp decline in this value along with a sharp increase in Memory\Pages/sec. With the other processes on the server possibly using parts of the system or file cache, there is always a balancing act trying to compete for memory on any given server. The good news is that Windows 2008 has made some significant changes in memory management to help prevent these system wide working set trims and to manage memory more efficiently. With server applications having the need for using more RAM as technology advances, it is imperative that memory management works effectively to prevent any negative performance problems on any given server. For more information on some of these memory management changes in Windows 2008, see http://www.microsoft.com/whdc/system/sysinternals/memmgt.mspx.

I hope this helps clarify some of the reasons why Exchange uses more memory overall and provide clarity for these most common questions.

A special thanks goes to Andrew Goodsell, Matt Gossage, and Ross Smith IV for their invaluable information in this area and Nick Basile for tech reviewing this document.

Mike Lagase
Support Escalation Engineer

Share this post :

Where does the time go? -519 Jet_errLogSequenceEnd - Part II

Exchange — Mon, 04 Aug 2008 20:00:00 GMT

In Part I of this blog post I covered the specifics of the -519 Jet_errLogSequenceEnd issue and the resolution. We had a great follow up question from our readers about the when and how often Exchange issues the ESE 514 warning events? We'll answer those questions in detail here in Part II.

How much time do I have?

The key information you should focus on in the ESE 514 event is the current log generation. Have a look at our TechNet document that explains this event (you'll notice this is for Exchange version 8.0 aka. Exchange 2007; it functions the same way for Exchange 6.5 and 6.0 or Exchange 2003 and 2000 respectively): http://www.microsoft.com/technet/support/ee/transform.aspx?ProdName=Exchange&ProdVer=8.0&EvtID=514&EvtSrc=ESE&LCID=1033

Here is the event string with variables:

Product:	Exchange
ID:	514
Source:	ESE
Version:	8.0
Symbolic Name:	ALMOST_OUT_OF_LOG_SEQUENCE_ID
Message:	%1 (%2) %3Log sequence numbers for this instance have almost been completely consumed. The current log generation is %4 which is approaching the maximum log generation of %5, there are %6 log generations left to be used. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated.

Here is what this looked like on an Exchange 2003 server in production (from a support case I recently worked on):

Event ID     : 514
Category     : None
Source       : ESE
Type         : Warning
Machine      : <Exchange Server Name>
Message      : Information Store (5144) <Store Name>: Log sequence numbers for this instance have almost been completely consumed. The current log generation is 1044490 (0x000FF00A) which is approaching the maximum log generation of 1048559 (0x000FFFEF), there are 4069 (0x00000FE5) log generations left to be used. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated.

Unfortunately these warnings were missed, and the server eventually reached:

Event ID     : 514
Category     : None
Source       : ESE
Type         : Warning
Machine      : <Exchange Server Name>
Message      : Information Store (5144) <Store Name>: Log sequence numbers for this instance have almost been completely consumed. The current log generation is 1048550 (0x000FFFE6) which is approaching the maximum log generation of 1048559 (0x000FFFEF), there are 9 (0x00000009) log generations left to be used. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated.

All the ESE 514 Events were missed and Exchange shut down the databases not long after the second event was logged. Recall from Part I of the blog that Exchange issues the MSExchangeIS 1159 error when transaction log sequence numbering has been exhausted.

What's interesting about these events is that Exchange tells you exactly how long you have before all log file generations are used up and we increase the frequency of warning messages the closer we get to Jet_errLogSequenceEnd.

When do the warnings start and how often do they occur?

For Exchange 2000, 2003, and 2007, we use thresholds to determine when and how often the ESE 514 warning events will be fired. Those thresholds are different for Exchange 2000/2003 and Exchange 2007 due to Exchange 2007's much longer transaction log sequence range. Recall from Part I of the blog the maximum transaction log sequence number in Exchange 2000/2003 is FFFF0 (1,048,560 decimal) while Exchange 2007 is 7fffffec (2,147,483,628 decimal).

The following sections describe the exact thresholds that trigger the first ESE 514 warning events and frequency at which subsequent warning events will be fired.

Exchange 2000/2003:

Exchange 2000/2003 will fire the first ESE 514 Event at log generation number 0xE0000 (917,504 decimal) for a given Storage Group. From that point, Exchange fires an ESE 514 warning event every 1000 logs until we hit log generation 0xFF000 (1,044,480 decimal). By this point Exchange is dangerously close to the maximum and goes into Panic Mode, issuing the Warning every 10 transaction logs until we run out.

Exchange 2007:

Exchange 2007 will fire the first ESE 514 Event at log generation number 0x7FFE0000 (2,147,352,576 decimal) for a given Storage Group. From that point, Exchange fires an ESE 514 warning event every 1000 logs until we hit log generation 0x7FFFF000 (2,147,479,552 decimal); at this point Exchange is dangerously close to the maximum and goes into Panic Mode, issuing the Warning every 10 transaction logs until we run out. If you indeed have run out of Exchange 2007 transaction logs - let us know. We'd love to hear how come that many logs are generated!

Conclusion:

The key to avoiding database shutdown due to -519 Jet_errLogSequenceEnd is to monitor your Exchange application logs. When you start to see ESE 514 Warnings make plans to reset transaction log sequence. If you're seeing those very frequently, better act fast.

Corbin Meek
Enterprise Communications Support

Share this post :

Avoiding Jitter: Jumpstarting the Exchange shell

Exchange — Fri, 01 Aug 2008 14:20:00 GMT

A little trick from Jeffrey Snover over on PowerShell team blog: Speeding Up PowerShell Startup. Running the script mentioned by Jeffrey speeds up PowerShell startup times. Yes, hard to believe at first, but after having run this on a few servers, as Jeffrey says - the reaction is "Wow!"

Paste the following in notepad and save it as Update-Gac.ps1 (or whatever you want to call it):

Set-Alias ngen @(
dir (join-path ${env:\windir} "Microsoft.NET\Framework64") ngen.exe -recurse |
sort -descending lastwritetime
)[0].fullName
[appdomain]::currentdomain.getassemblies() | %{ngen $_.location}

Note: On x86 systems; replace Framework64 in the second line of this script with Framework.
[Optional] Close all open windows
[Optional] Start the Exchange Management Shell and note the time it takes to start up
Run the script: .\Update-Gac.ps1 (or whatever you saved it as)
Quit all open windows, start the Shell. Notice the difference?

As Jeffrey says in his blog post, the PowerShell team had a problem in Windows PowerShell v1, that caused the binaries not to be NGENed.

What's NGEN?
Managed code exists in Common Intermediate Language (CIL). The Just In Time (JIT) compiler takes that and converts it into native code for a particular processor/platform - a process also referred to as "jitter". If you don't want this translation to happen at runtime, you can use NGEN.exe - the Native Image Generator - to convert managed code from the CIL to native code. Native code does not need to be compiled at run time, and thereby avoids jitter.

So how does speeding up PowerShell help Exchange?
The Shell is built on Windows PowerShell technology, and under the hood, the Exchange Management Console also executes PowerShell cmdlets. If you're running Exchange Server 2007 on Windows Server 2003, you're likely to see performance gains in shell start-up times after running this script. On one of my Exchange 2007 servers, EMS now starts up in as little as 2-3 seconds. Disclaimer: Your mileage may vary.

Feel free to leave feedback about results you're seeing with this.

- Bharat Suneja

Share this post :