While recently attending TEC2010 in Los Angeles, I was asked what do I think of deploying Exchange 2010 using JBOD and why do I need to use network attached storage? After discussing the topic in length with both attendees and presenters it became clear that there was a misunderstanding of what JBOD means for Exchange 2010. Simply put, JBOD means that you will not be using the native RAID features of your storage platform and instead rely on Exchange 2010 to manage individual database protection. Ergo, one database and log set per disk. With JBOD, a disk failure equals a database failover which equals a reseed.
Microsoft’s targeted I/O ranges for Exchange 2010 were driven by the goal of deploying Exchange 2010 on cheaper, lower I/O, higher capacity 7.2k SATA disks. The concept of one database and log set per disk sounds promising providing you are deploying your architecture in line with Microsoft’s supported recommendation of managing at least three copies of each database. For more details on Exchange 2010’s mailbox storage requirements click here.
A typical enterprise class 2TB 3G 7.2k 3.5” SATA drive retails for around $950. (Bulk and vendor discounts are not applied) That will equal a rough disk cost estimate of $2850 per database for a supported JBOD architecture. We’ will need a minimum of three mailbox servers with three DAS arrays so we don’t have a single point of failure. With Exchange 2010 we can now combine the CAS, HUB and Mailbox role onto a single server to keep server hardware and licensing costs down. Be aware that if the CAS and the Mailbox role reside on the same server and said server is a member of a DAG Windows Network Load Balancing will not be available. A hardware load balancer will be required to achieve true high availability among the Client Access services.
Using Microsoft’s Mailbox role storage calculator for Exchange 2010 with the following data let’s examine the results.
- User count: 1000
- Disk specs: 2TB 7.2k SATA 3.5” – ~$950 each
- Mailbox Servers: 3
- DAG: 1
- User profile: 250 message per day (your mileage will vary)
- Mailbox Size: 5GB (most companies will choose smaller values)
- Multi-role: Yes
- Lagged Copies: 0
- HA: Yes
- Site Resiliency: No
Reviewing the JBOD storage recommendations we are looking at six disks for the databases at a cost of $2850 per database. Total disk cost for all servers is ~$17,100. Each disk is only being utilized up to 1.6TB effectively losing 20% of total disk capacity. At only 6 disks per mailbox server, there are a number of DAS storage options available. There are a number of considerations we are not factoring into our calculations such as backup strategy, I/O requirements, preferred vendors, rack and power requirements, etc… I want to keep this example as simple as possible. Microsoft recommends that if you want to go backup-less you will need at least one lagged copy in your environment.
Scale things up a bit to 10,000 users and the results are dramatic.
Disk count has now increased by 95% from six disks per server to 57, or 171 disks total. Total disk cost is now at ~$163,000. Now considering that each Mailbox server will need at least 57 disks the DAS vendor options are reduced. Doing a quick search on the Internet shows a typical DAS unit retails for around $75k with support for up to 70 2TB disks. (your mileage will vary)
For a 10,000 user environment we are now at total cost of ~$225,000 for the DAS array with disks. Since we are only utilizing ~80% percent of the total capacity of the DAS, we are wasting about $45,000 of our initial investment. Pause for a moment here and think of the overhead involved with administrating 171 database copies. Since the storage device is dedicated to Exchange we will be unable to allocate the unused 20% of capacity to other applications in our environment but, at least we have room for growth.
At 20,000 users (all things still being equal) three Mailbox servers isn’t going to support the environment. Two more mailbox servers will need to be added along with two more DAS arrays to support the solution. Disk count increased to 66 per server for a total of 330 disks or databases. That’s an increase of 48%. Now what if you don’t want to go with the 2TB disks and only can afford a 1TB disk? Add two more mailbox servers for a total of seven and we are now at 90 disks per server or 630 disks/databases total. I’m not going to even do the math, you get the point.
I hope this has helped illustrate the often overlooked complexities involved in deploying JBOD with DAS and the extra overhead that comes with scaling the solution. Exchange administrators are very excited about the new JBOD option but do your homework, weigh all the options, talk to your storage vendors or consult a professional consulting services organization for planning assistance. In the end, the initial investment will be well worth it in the long run. More often your specific business needs will be better served investing in a SAN or NAS device where the storage services can be shared among servers and the technologies of thin provisioning and tiered storage can be implemented. As this white paper illustrates, your initial investment in a SAN/NAS solution will provide a better TCO overall and a better quality of life for your Exchange administrators.
In our 10,000 user example introducing RAID would reduces our database count by 47% to 36 while also allowing a reduction in the number of database copies to two. Disk count increases 29% from 57 to 80 but, let’s face it, not all users in your environment have (or require) 5 GB of mail data. Keep in mind the mailbox role calculator designs the storage for 120% of mailbox capacity. Realistically you would design a tiered model at around 1 GB for the general population with a small portion of the staff getting a larger mailbox capacity. Or maybe you want everyone to have higher a mailbox capacity and import the user’s PSTs into their mailbox and eliminating the use of PSTs altogether. With Exchange 2010, Microsoft now considers a mailbox above 10GB to be “large.” Considering all the benefits around TCO, capacity utilization, scalability, management, HA, DR, Backup and Recovery and Server Virtualization…is DAS the right choice for you? Exchange is often a business critical application and the underlying infrastructure should reflect that.
But don’t just take my word for it. Here are some articles worth reading.
- From Whence Redundancy? Exchange 2010 Storage Essays, part 1
- The Disk’s The Thing! Exchange 2010 Storage Essays, part 2
- FAS, the new DAS – Using FAS in a DAS configuration for Exchange
- Why Microsoft’s Head is up its DAS
- TechEd Online – Storage in Microsoft Exchange Server 2010
- TechNet Webast – How Microsoft IT Deployed Exchange 2010 on Premises
- Exchange SAN vs. DAS – The Saga Continues… – Added 1/21/2011
- Robert’s Rules of Exchange: Storage Planning & Testing – Added 1/21/2011