Techdocs Library > White papers > Positioning Solid State Disk (SSD) in an AIX environment

Document Author:	Dan Braden		Document ID:	WP101560
Doc. Organization:	Advanced Technical Support		Document Revised:	10/09/2009
Product(s) covered:	Power 6

Abstract: This document helps customers determine if they should purchase solid state disk (SSD) technology for AIX systems by examining performance data and looking at costs.

Positioning Solid State Disk (SSD) in an AIX environment This article will help you determine if you should use SSDs instead of hard disk drives (HDDs) for your application. Today, especially for random IO applications, one sizes disk based on IOPS (aka. IOs per second), not on space. So one must purchase enough HDDs to meet the performance needs. So it becomes reasonable to compare the cost per IO of SSDs vs HDDs, and on this measure SSDs are much less expensive. However, SSDs don't have the capacity that HDDs do. So this article shows you how to compare the costs of HDDs vs. SSDs, determine access density of your data, and determine if it's less expensive to purchase SSDs rather than HDDs for a specific configuration. Customers who will find SSDs beneficial, will fall into three categories: Those who will find them less expensive than HDDs Those who need more performance than their HDDs can provide them today Those currently using RAM file systems Given the high IOPS that a single SSD can sustain, it's possible that a one or a few SSDs can be used instead of many HDDs. Depending on R/W ratio and assuming an IO size of 4 KB, a single SSD can perform somewhere around 14,000 to 29,000 IOPS, as compared to perhaps 175 IOPS for a single HDD. So theoretically, one SSD could replace up to 165 HDDs. However, given that a single SSD has only 69 GB of space, it doesn't have near the storage capacity of even a single typical HDD available today. So the ability to actually replace HDDs with SSDs depends on the ability to find data that has high IOPS requirements and will fit on the SSDs. This characteristic of data is known as access density, or IOPS/GB. However, given that a single SSD currently lists for $10,000 and a single HDD lists for $498, one would only need to replace 20 HDDs. However there are other costs involved including, disk enclosures, disk/SSD adapters, power, cooling, maintenance and space. An exact configuration would be required for an exact estimate of the costs. A reasonable estimate of the power cost for a HDD is about $25/year including the power for the enclosure. The power cost for a SSD is about half the cost of a 15 K RPM HDD, or about the cost of a SFF HDD. One SSD vendor publishes a figure of 2,970 IOPS/Watt for SSDs vs 17.5 IOPS/Watt for 15 K RPM HDDs (http://www.stec-inc.com/downloads/whitepapers/Performance_Power_Advantages.pdf). This doesn't include cooling cost savings for the data center which may double the electric cost savings. Nor does it include savings for floor space. There is another potential benefit that is more difficult to quantify. And that is in performance gains. Writes to a SSD will take about as long as writes to disk subsystem cache. Without write cache HDDs will be much slower than SSDs. Some SAS adapters have write cache (writes to adapter cache are slightly faster than directly to SSD). But you won't see the same amazing difference in SSD vs HDD performance with writes as with random reads. Reads to SSDs take about 0.33 ms versus somewhere around 4 to 12 ms for HDDs depending on how far the actuator has to seek, how much the disk has to rotate to the right track and how busy the disk is. This translates into higher maximum system thruput, better application response times, and faster batch job run times, potentially much faster. Actual application performance benefits are best determined by an actual benchmark, and the value to the customer will vary. The paper at ftp://ftp.software.ibm.com/common/ssi/sa/wh/n/pow03025usen/POW03025USEN.PDF provides an example for SAP on DB2 where there are significant improvements in performance. So SSDs are best used in high access density environments. Access density is IOPS/GB, so high access density environments mean we do a lot of IOs over a relatively small amount of data. This often applies to database indexes. And the higher the proportion reads, the greater the performance benefits. SSDs can also be used in clustered applications where they need to be connected to more than one system. IBM currently offers SSDs in the DS8000, or via SAS adapters on POWER6 systems. A pair of PCIe SAS RAID adapters, FC 5903, allows one to attach SSDs in an EXP 12S SAS drawer. Some customers will find that disks are just too slow to achieve the performance they need. This kind of environment is characterized by the inability of the application to break the job into multiple threads and allow them to run simultaneously, usually due to locking of data on disk. System performance statistics will likely show the CPU waiting on IO (e.g., %iowait > 0 and usually much greater). Since we can read data from SSDs in less than 1 ms, while reading the same data from disk can take up to 15 ms (or worse if IOs are queued for the disk), using SSD can significantly improve performance. This results in the ability of a system to do more transactions per second, or for batch jobs to run much faster assuming we're doing random reads from storage. Customers using RAM file systems may find that replacing expensive system memory with SSDs to be cost effective. The cost of memory varies considerably but assuming a cost of $500/GB, system RAM costs more than 3X that of SSDs. The time to do an IO, or latency, is much slower with SSDs as compared to RAM, so there's a performance cost trade off. SSDs do have an advantage of saving the data when the system is shut down, in the event of a system crash or power outage. The best way to determine the actual impact on application performance by using SSDs is to do a benchmark. So assuming you want to determine, on a cost basis only, whether SSDs fit in your environment, we have to look at the configuration, the IO environment and do some calculations. RAID Many customers use RAID to ensure the application can keep running in case of a disk failure, or to ensure the recoverability of data. So this affects cost of the HDD or SSD solution. While the reliability of SSDs is greater than HDDs (HDDs are after all, mechanical), SSDs do wear out over time, so we generally recommend using RAID to protect your data. A SSD memory cell can only be written to a limited number of times (this varies depending among SSD technology, but is around 1,000,000 times per memory cell). SSDs do have technology that helps lengthen their life: wear leveling, over provisioning and bad block relocation. RAID is often used for HDDs. RAID also affects the ratio of physical IOs to application IOs depending on the R/W ratio and the RAID level used. Writes to RAID 5 arrays require 4 physical IOs (two read and two writes) while writes to RAID 1 or 10 arrays requires 2 physical writes. Since AIX IO statistics measure application IOs, the use of RAID affects calculations on how many HDDs are needed for a given IOPS and R/W ratio. Use these formulas to determine potential application IOPS given a configuration, or number of SSDs or HDDs for a given IO environment: N = number of SSDs R = proportion of IOs that are reads W = 1-R = proportion of IOs that are writes D = IOPS for a single SSD (use 16,000 to be conservative unless all R or W, and for HDDs use 175) JBOD or RAID 0: IOPS bandwidth = NxD N = IOPS/D RAID 1 or RAID 10: IOPS bandwidth = NxD/(R+2W) N = IOPSx(R+2W)/D RAID 5: IOPS bandwidth = NxD/(R+4W) N = IOPSx(R+4W)/D RAID 6: IOPS bandwidth = NxD/(R+6W) N = IOPSx(R+6W)/D Note that 175 IOPS per 15 K RPM HDD is somewhat arbitrary. As IOPS increase for a HDD, IO service times start increasing as the IOs increasingly have to wait for other IOs to complete before they can be serviced. The higher value IOPS have worse IO service times, while lower values have better IO service times. Poorer IO service times reduce maximum system thruput and lengthen batch job run times. Higher RPM disks can perform more IOPS than lower RPM disks. The IO environment Customers who will find SSDs less expensive than HDDs will also find they have high access density data. Access density for your data can be determined from the AIX iostat, lvmstat or filemon commands. The filemon command is a trace and has significant overhead, so isn't the best tool for long term data collection. Nevertheless it can also provide useful information regarding if the IO is sequential or random for reads and or writes, and average IO sizes. The iostat command shows IO statistics to hdisks. For long term data collection to examine rates to LVM Logical Volumes (LVs), the tool to use is lvmstat. Long term data collection tools are important to find peak IOPS intervals which are used to size the number of HDDs or SSDs needed. It is difficult to collecting filemon data during the peak interval since we don't necessarily know when the peak interval is. One must enable collection of lvmstat data for a VG via the following command: # lvmstat –e –v <vgname> Then one can collect statistics for LVs or LPs (logical partitions) in the VG: ... The first command above shows the iocnt which is the number of IOs in the interval which in this case is 3 seconds. The second command shows iocnt for individual partitions in lv1. Knowing the size of the LV, we can calculate the access density. We can get the size of the LV using the # lslv <LVname> command, e.g.: In this case the lv1 LV is 719x4=2876 MB. The second command shows the iocnt for each logical partition, and the Log_part refers to the logical partition number. The first report for lvmstat shows statistics since the system was booted. So we'll normally ignore the first report and instead look at statistics over normal production periods. And since storage is sized for peak IOPS workloads, that would also be used for SSDs. So one would find the interval with the peak IOPS workload, and examine the access density for each LV, and potentially each LP. The procedure to determine the data would be to first determine the peak IOPS period. Then for that period, one would want to collect lvmstat data for each VG and find LVs with high access density. Or collect lvmstat data for all LPs and sort them based on access density. One would use LVM migration to add SSD disks to a VG, then either the migratepv command to move LVs to the SSDs, or the migratelp command to move individual LPs from HDDs to SSDs. HDD versus SSD costs We'll work thru a few examples and only consider the cost of the SSDs vs HDDs, including adapter costs, enclosure costs and maintenance but not including cooling or space costs. We won't include power costs of SSDs but the cooling savings are likely to be much higher. Nor will we consider any cost benefits from the performance improvements we'll get. And we won't consider the costs of hot spare disks in a RAID configuration. We'll use the list price of a SSD at $10,000 and a HDD at $498. Prices in this document are list prices at the time of the writing and will change over time, so your numbers may vary. But the analysis will be useful for comparison and for understanding the methodology. We'll use a net present value cost of $75 for power for each HDD: this is assuming 4 years of power at $24 each year discounted at a rate of 10%. Monthly maintenance costs for a SSD are $60, and calculating net present value at a rate of 10% for 4 years is about $2282. Monthly maintenance for a EXP 12S SAS disk drawer that holds 12 SAS disks is $133, and discounted at 10% over 4 years results in a net present value cost of about $422 for maintenance for each HDD. We can attach up to 8 SSDs in the CEC, or for a larger number we might use a EXP 12S SAS drawer. Actual configurations will vary, but we'll assume SSDs are placed in the CEC. We'll assume the use of the planar integrated SAS adapter. The planar integrated SAS adapters support RAID 0, and with the SAS RAID enablement card ($2,500) supports RAID 5, 6 and 10 and provides write cache. Non-RAID SAS adapters are advertised as supporting RAID 0 and 10 (note that 2 disk RAID 10 is RAID 1); however, they also support RAID 5 and 6 but this fact is not advertised as there's a significant performance penalty for random writes and isn't generally recommended. To attach a bunch of SAS HDDs, we'll use the EXP 12S SAS drawer ($4,500) which has 12 disk bays. We can attach 4 drawers to an adapter which also requires 2 YO cables ($90 each) and 4 EE cables ($70 each). We'll assume the use of a PCIe Dual - x4 3Gb SAS RAID Adapter ($2199). So the adapter, drawer and cables for 48 disks totals This adds a cost of $385 per HDD. Adapter 2,999 4 SAS drawers 18,000 Cables 460 Total 21,459 Per HDD=$447 for the HDD infrastructure So HDDs cost: List Price 498 Infrastructure 447 Maintenance 422 Power 75 Total 1,442 While SSDs cost: List Price 10,000 Maintenance 2,282 SAS RAID Enablement Card 2,500 Total 14,782 So to be cost effective, for every SSD we purchase, we need to replace 14782/1442 = 10.25 HDDs. Customers may also consider replacing HDDs in a disk subsystem with SSDs. So for comparison, we'll use this DS4800 configuration: HDD costs in a disk subsystem DS4800 Model 80 with 4 GB of cache, 150 73.4 GB 15K RPM disks in 10 EXP810 enclosures. List price is $337,298 plus about $5,000 for two fibre channel adapters with an annual maintenance cost of $24,020. Net present value for maintenance at 4 years, discounted at 10%, is $76,140. This works out to a cost per HDD of $2,790. Comparing SSDs to HDDs in a disk subsystem isn't exactly an apples to oranges comparison. The DS4800 supports making multiple LUNs from a single RAID array, the ability to assign storage to multiple servers, remote mirroring to another DS4800, dynamic volume expansion, dynamic array expansion, more intelligent cache and other capabilities. Configurations will vary so you may want to compare your configuration, or the alternative configuration you're considering to SSDs. SSD costs in the DS8000 The DS8000 offers 73 GB and 146 GB via a RPQ. The SSDs are ordered in sets of 16: FC 6016 for sixteen 73 GB SSDs ($910,000) or FC 6116 for sixteen 146 GB SSDs ($1,400,000). These are configured into 7+P RAID 5 arrays. Maintenance for each feature is $5500 or $9600 respectively; however, that's after a 4 year warranty, so we'll ignore those costs, along with power, and infrastructure. Costs are: AIX SAS SSDs 69 GB $10,000 $145/GB DS8000 73 GB SSDs $910,000 for 16 $780/GB 146 GB SSDs $1,400,000 for 16 $599/GB So SSDs cost substantially more in the DS8000. There is more functionality: the ability to assign small LUNs to a server, the ability to assign multiple LUNs to a server or to multiple servers, the ability to create FlashCopies of the LUNs (potentially to lower performing storage), intelligent cache, higher levels of support, etc. So it's probably better to compare DS8000 SSDs to DS8000 HDDs. A drive set (16 HDDs) of 146 GB 15 K RPM disks lists for $48,500, or $3031 per HDD, not including any maintenance. Assuming the HDDs can perform 175 IOPS and the SSDs can perform 17,500 IOPS, one SSD can replace up to 100 HDDs provided the data will fit on it. From a cost standpoint, a DS8000 73 GB SDD costs about as much as 19 HDDs, while a 146 GB SDD costs about as much as 29 HDDs. A few examples SSDs replacing HDDs in a non-protected configuration This scenario would apply to customers using RAM file systems, or when we can lose current data and restore from a backup in the event of a HDD or SSD failure. Since we need to replace 10.25 HDDs for each SSD, the HDDs would have the following IOPS bandwidth: 10.25 x 175 = 1794 IOPS. Since each SSD has 69 GB of space, we'd need data with an access density of 1794/69 = 26 IOPS. So if we had 69 GB of data with that access density we could use 1 SSD, or with 138 GB we'd get two SSDs, etc. Two SSDs replacing HDDs in a RAID 10 configuration Consider the case where we use 2 SSDs in a RAID 1 configuration in lieu of HDDs. From a cost standpoint, we have to replace 21 HDDs and we can only put 69 GB on data on the SSDs. Let's further assume that we're replacing HDDs in a RAID 10 configuration, and the IO environment is 80% random reads. We can't put 21 disks into a RAID 10 configuration but we'll use that number for our calculations. The HDDs have an application IOPS bandwidth of 175 x 21 / (0.8+ 2 x 0.2) = 3063 IOPS. One SSD can easily do this many IOPS. So if we can find 69 GB of data with at least 3063 IOPS, or an access density of at least 3063/69 = 44 IOPS/GB, then we can cost justify the purchase. Three SSDs replacing HDDs in a RAID 5 configuration Consider the case where we use 3 SSDs in a RAID 5 configuration in lieu of HDDs. Here we need to replace 32 HDDs and will have 138 GB of protected space. Here we'll assume we're replacing HDDs in a RAID 5 configuration and the IO environment is 70% random reads. The HDDs have an application IOPS bandwidth of 175 x 32 / (0.7 + 4 x 0.3) = 2947 IOPS. So we need data with an access density of at least 2947/138 = 21 IOPS/GB. Eight SSDs replacing HDDs in a RAID 5 configuration Finally, consider the case where we use 8 SSDs in a RAID 5 configuration. We have to replace 82 HDDs. We'll assume 80% reads. The HDDs have an application IOPS bandwidth of 175 x 82 / (0.8 + 4 x 0.2) = 8969 IOPS. We'll need data with an access density of at least 8969 / (69 x 7) = 19 IOPS/GB. Final thoughts Given sufficient access density with enough data, SSDs can be a quantifiable cost effective alternative to HDDs. They can also provide significant performance benefits which also are of benefit, though how much will depend on your situation. Long term, the price of SSDs will continue to drop and they will become more prevalent. There is also the choice of purchasing SSDs directly attached to servers, or those in disk subsystems such as the DS8000.

Classification:	Hardware	Category:	Product Positioning
Platform(s):	IBM Power Systems; IBM System Storage
O/S:	AIX	Keywords:	SSD, AIX