# Amazon EBS volume performance Several factors, including I/O characteristics and the configuration of your instances and volumes, can affect the performance of Amazon EBS. If you follow the guidance on our Amazon EBS and Amazon EC2 product detail pages you'll usually achieve good performance. However, there are some cases where you might need to do some tuning to achieve peak performance. We recommend that you tune performance with information from your actual workload, in addition to benchmarking, to determine your optimal configuration. After you learn the basics of working with EBS volumes, it's a good idea to look at the I/O performance you require and at your options for increasing Amazon EBS performance to meet those requirements. AWS updates to the performance of EBS volume types might not immediately take effect on your existing volumes. To see full performance on an older volume, you might first need to perform a `ModifyVolume` action on it. For more information, see [Modify an Amazon EBS volume using Elastic Volumes operations](ebs-modify-volume.md). **Topics** + [ ## Amazon EBS performance tips ](#tips) + [ # Amazon EBS optimization ](ebs-optimization.md) + [ # Initialize Amazon EBS volumes ](initalize-volume.md) + [ # Configurable instance bandwidth weighting ](instance-bandwidth-configuration.md) + [ # Amazon EBS I/O characteristics and monitoring ](ebs-io-characteristics.md) + [ # Amazon EBS and RAID configuration ](raid-config.md) + [ # Benchmark Amazon EBS volumes ](benchmark_procedures.md) ## Amazon EBS performance tips These tips represent best practices for getting optimal performance from your EBS volumes in a variety of user scenarios. ### Use EBS-optimized instances On instances without support for EBS-optimized throughput, network traffic can contend with traffic between your instance and your EBS volumes; on EBS-optimized instances, the two types of traffic are kept separate. Some EBS-optimized instance configurations incur an extra cost (such as C3, R3, and M3), while others are always EBS-optimized at no extra cost (such as M4, C4, C5, and D2). For more information, see [Amazon EBS optimization](ebs-optimization.md). ### Configure instance bandwidth For supported instance types, you can configure the instance bandwidth weighting to increase Amazon EBS bandwidth by 25 percent using the `ebs-1` bandwidth weighting. This feature allows you to optimize your instance's network resource allocation between EBS and VPC networking, potentially improving EBS performance for I/O-intensive workloads. For more information, see [Configurable instance bandwidth weighting](instance-bandwidth-configuration.md). ### Understand how performance is calculated When you measure the performance of your EBS volumes, it is important to understand the units of measure involved and how performance is calculated. For more information, see [Amazon EBS I/O characteristics and monitoring](ebs-io-characteristics.md). ### Understand your workload There is a relationship between the maximum performance of your EBS volumes, the size and number of I/O operations, and the time it takes for each action to complete. Each of these factors (performance, I/O, and latency) affects the others, and different applications are more sensitive to one factor or another. For more information, see [Benchmark Amazon EBS volumes](benchmark_procedures.md). ### Be aware of the performance penalty When initializing volumes from snapshots There is a significant increase in latency when you first access each block of data on a new EBS volume that was created from a snapshot. You can avoid this performance hit using one of the following options: + Access each block prior to putting the volume into production. This process is called *initialization* (formerly known as pre-warming). For more information, see [Manually initialize the volumes after creation](initalize-volume.md#ebs-initialize). + Enable fast snapshot restore on a snapshot to ensure that the EBS volumes created from it are fully-initialized at creation and instantly deliver all of their provisioned performance. For more information, see [Amazon EBS fast snapshot restore](ebs-fast-snapshot-restore.md). ### Factors that can degrade HDD performance When you create a snapshot of a Throughput Optimized HDD (`st1`) or Cold HDD (`sc1`) volume, performance may drop as far as the volume's baseline value while the snapshot is in progress. This behavior is specific to these volume types. Other factors that can limit performance include driving more throughput than the instance can support, the performance penalty encountered while initializing volumes created from a snapshot, and excessive amounts of small, random I/O on the volume. For more information about calculating throughput for HDD volumes, see [Amazon EBS volume types](ebs-volume-types.md). Your performance can also be impacted if your application isn’t sending enough I/O requests. This can be monitored by looking at your volume’s queue length and I/O size. The queue length is the number of pending I/O requests from your application to your volume. For maximum consistency, HDD-backed volumes must maintain a queue length (rounded to the nearest whole number) of 4 or more when performing 1 MiB sequential I/O. For more information about ensuring consistent performance of your volumes, see [Amazon EBS I/O characteristics and monitoring](ebs-io-characteristics.md) ### Increase read-ahead for high-throughput, read-heavy workloads on `st1` and `sc1` (*Linux instances only*) Some workloads are read-heavy and access the block device through the operating system page cache (for example, from a file system). In this case, to achieve the maximum throughput, we recommend that you configure the read-ahead setting to 1 MiB. This is a per-block-device setting that should only be applied to your HDD volumes. To examine the current value of read-ahead for your block devices, use the following command: ``` $ sudo blockdev --report /dev/ ``` Block device information is returned in the following format: ``` RO RA SSZ BSZ StartSec Size Device rw 256 512 4096 4096 8587820544 /dev/ ``` The device shown reports a read-ahead value of 256 (the default). Multiply this number by the sector size (512 bytes) to obtain the size of the read-ahead buffer, which in this case is 128 KiB. To set the buffer value to 1 MiB, use the following command: ``` $ sudo blockdev --setra 2048 /dev/ ``` Verify that the read-ahead setting now displays 2,048 by running the first command again. Only use this setting when your workload consists of large, sequential I/Os. If it consists mostly of small, random I/Os, this setting will actually degrade your performance. In general, if your workload consists mostly of small or random I/Os, you should consider using a General Purpose SSD (`gp2` and `gp3`) volume rather than an `st1` or `sc1` volume. ### Use a modern Linux kernel (*Linux instances only*) Use a modern Linux kernel with support for indirect descriptors. Any Linux kernel 3.8 and above has this support, as well as any current-generation EC2 instance. If your average I/O size is at or near 44 KiB, you may be using an instance or kernel without support for indirect descriptors. For information about deriving the average I/O size from Amazon CloudWatch metrics, see [Amazon EBS I/O characteristics and monitoring](ebs-io-characteristics.md). To achieve maximum throughput on `st1` or `sc1` volumes, we recommend applying a value of 256 to the `xen_blkfront.max` parameter (for Linux kernel versions below 4.6) or the `xen_blkfront.max_indirect_segments` parameter (for Linux kernel version 4.6 and above). The appropriate parameter can be set in your OS boot command line. For example, in an Amazon Linux AMI with an earlier kernel, you can add it to the end of the kernel line in the GRUB configuration found in `/boot/grub/menu.lst`: ``` kernel /boot/vmlinuz-4.4.5-15.26.amzn1.x86_64 root=LABEL=/ console=ttyS0 xen_blkfront.max=256 ``` For a later kernel, the command would be similar to the following: ``` kernel /boot/vmlinuz-4.9.20-11.31.amzn1.x86_64 root=LABEL=/ console=tty1 console=ttyS0 xen_blkfront.max_indirect_segments=256 ``` Reboot your instance for this setting to take effect. For more information, see [Configure GRUB for paravirtual AMIs](https://docs.aws.amazon.com/linux/al2/ug/UserProvidedKernels.html#configuringGRUB). Other Linux distributions, especially those that do not use the GRUB boot loader, may require a different approach to adjusting the kernel parameters. For more information about EBS I/O characteristics, see the [Amazon EBS: Designing for Performance](https://www.youtube.com/watch?v=2wKgha8CZ_w) re:Invent presentation on this topic. ### Use RAID 0 to maximize utilization of instance resources Some instance types can drive more I/O throughput than what you can provision for a single EBS volume. You can join multiple volumes together in a RAID 0 configuration to use the available bandwidth for these instances. For more information, see [Amazon EBS and RAID configuration](raid-config.md). ### Monitor Amazon EBS volume performance You can monitor and analyze the performance of your Amazon EBS volumes using Amazon CloudWatch, status checks, and EBS detailed performance statistics. For more information, see [Amazon CloudWatch metrics for Amazon EBS](using_cloudwatch_ebs.md) and [Amazon EBS detailed performance statistics](nvme-detailed-performance-stats.md). # Amazon EBS optimization An Amazon EBS–optimized instance uses an optimized configuration stack and provides additional, dedicated capacity for Amazon EBS I/O. This optimization provides the best performance for your EBS volumes by minimizing contention between Amazon EBS I/O and other traffic from your instance. EBS–optimized instances deliver dedicated bandwidth to Amazon EBS. When attached to an EBS–optimized instance: + `io2` Block Express volumes are designed to deliver an average latency of under 500 microseconds for 16KiB I/O operations. `io2` Block Express volumes also deliver better outlier latency compared to General Purpose volumes, reducing the frequency of I/Os exceeding 800 microseconds by over 10 times. `io1` and `io2` volumes are designed to deliver at least 90% of their provisioned IOPS performance 99.9% of the time in a given year. + `gp2` and `gp3` volumes are designed to deliver at least 90% of their provisioned IOPS performance 99% of the time in a given year. + `st1` and `sc1` volumes deliver at least 90% of their expected throughput performance 99% of the time in a given year. Non-compliant periods are approximately uniformly distributed, targeting 99% of expected total throughput each hour. For more information, see [Amazon EBS volume types](ebs-volume-types.md). The I/O size of your workload will impact the observed average latency as latency increases with larger I/O size. For example, `io2` Block Express volumes are designed to deliver an average latency of under 500 microseconds for 16KiB I/O operations. For more information, see [Amazon EBS–optimized instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-optimized.html) in the *Amazon EC2 User Guide*. # Initialize Amazon EBS volumes When you create an Amazon EBS volume, either from an EBS snapshot or from another EBS volume (volume copy), the data blocks must be written to the volume before you can access them. For volumes created from snapshots, the data blocks must be downloaded from Amazon S3 to the new volume. For volume copies, the data blocks must be copied from the source volume to the volume copy. This process is called *volume initialization*. During this time, the volume being initialized might experience increased I/O latency and decreased performance. Full volume performance is achieved only once all storage blocks have been downloaded and written to the volume. **Note** Empty volumes deliver their maximum performance immediately after creation and do not require initialization. The default volume initialization rate fluctuates throughout the initialization process, which could make completion times unpredictable. To minimize the performance impacts associated with volume initialization, you could use the following options: **Note** Volume initialization rate and fast snapshot restore are not supported for volume copies. For more information, see [Volume copy initialization](ebs-copying-volume.md#copy-volume-initialization). **Topics** + [ ## Use an Amazon EBS Provisioned Rate for Volume Initialization ](#volume-initialization-rate) + [ ## Use a snapshot that is enabled for fast snapshot restore ](#volume-initialization-fsr) + [Manually initialize volumes](#ebs-initialize) + [Monitor volume initialization](ebs-initialize-monitor.md) ## Use an Amazon EBS Provisioned Rate for Volume Initialization When you create an Amazon EBS volume from a snapshot, you can optionally specify an Amazon EBS Provisioned Rate for Volume Initialization (volume initialization rate) that ranges from 100 to 300 MiB/s. If you specify a volume initialization rate, the snapshot blocks are downloaded from Amazon S3 and written to the volume at the specified rate after creation. This enables you to create volumes that become fully initialized and fully performant in a predictable amount of time. Using a volume initialization rate is especially useful when you are creating multiple volumes simultaneously and you need all of them to be initialized in a predictable amount of time. **Note** Amazon EBS Provisioned Rate for Volume Initialization is supported with all Amazon EBS volume types, and all Amazon EC2 instance types, including Amazon EC2 Mac instances. You can specify a volume initialization rate: + For individual volume creation requests + For EBS volume block device mappings in instance launch requests + For EBS volume block device mappings in launch templates + For EBS volumes created by root volume replacement tasks + For EBS volumes on Amazon EKS clusters (created by EBS CSI Driver) and Amazon ECS clusters **Topics** + [ ### How it works ](#consistent-rate-how) + [ ### Considerations ](#consistent-rate-considerations) + [ ### Quotas ](#consistent-rate-quota) + [ ### Billing ](#consistent-rate-billing) ### How it works When you create a volume with a volume initialization rate, the snapshot blocks are downloaded from Amazon S3 to the volume at the rate you specify. The amount of time taken to initialize the volume depends on the following: + The size of the snapshot data, not the size of the volume being created. **Tip** To find a snapshot's data size, check the `FullSnapshotSizeInBytes` field in the [ describe-snapshots](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-snapshots.html) command output, or the **Full snapshot size** field in the console. + The volume initialization rate that you specify For example, if you create a 20 GiB volume using a snapshot that has 10 GiB of data, and you specify a volume initialization rate of 300 MiB/s, the volume will be fully initialized in approximately 34.1 seconds (10 GiB / 300 MiB/s = 34.1 seconds). Similarly, if you create 10 volumes with that same snapshot and volume initialization rate concurrently, all 10 volumes will be fully initialized in 34.1 seconds. ### Considerations + You can specify a volume initialization rate of between 100 and 300 MiB/s. + When you specify a volume initialization rate, the charges and completion time are based on the size of the snapshot data (not the size of the volume) and the rate you specify. For more information, see [Billing](#consistent-rate-billing). + Amazon EBS delivers an average rate that is within 10 percent of the volume initialization rate that you specify for 99 percent of the time. + If you specify a volume initialization rate and use a snapshot that is enabled for fast snapshot restore, Amazon EBS uses the specified rate instead of fast snapshot restore. To use fast snapshot restore instead, do not specify a volume initialization rate. + If Amazon EBS can't initialize the volume at the specified volume initialization rate due to capacity constraints or because you have exceeded your [ quota](#consistent-rate-quota), the request fails. + You can't specify a volume initialization rate for volumes created on AWS Outposts, or in Local Zones or Wavelength Zones. ### Quotas There is a limit of 5,000 MiB/s on the cumulative volume initialization rate that you can request across concurrent volume creation requests. For example, you can make 50 concurrent volume creation requests with a rate of 100 MiB/s (50 simultaneous requests \$1 100 MiB/s rate), or 25 concurrent requests with a rate of 200 MiB/s (25 simultaneous requests \$1 200 MiB/s rate). This limit applies on a per Region basis. If a request exceeds this limit, it fails. Either wait for some of the in-progress requests to complete or request a quota increase. For more information, see [Quotas for Amazon EBS](ebs-resource-quotas.md). ### Billing When you create a volume with a volume initialization rate, you are charged a rate per GiB of snapshot data, per MiB of specified initialization rate. The rate varies by Region. For more information, see [Amazon EBS pricing](https://aws.amazon.com/ebs/pricing/). You are charged based on the size of the snapshot data, not the size of the volume. For example, if you create a snapshot of a volume that is 100 GiB in size, but has only 50 GiB of data, the snapshot has a volume size of 100 GiB, but the snapshot data size is 50 GiB. If you use that snapshot to create a volume and specify a volume initialization rate, your charges are based on the 50 GiB of snapshot data. **Tip** To find a snapshot's data size, check the `FullSnapshotSizeInBytes` field in the [ describe-snapshots](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-snapshots.html) command output, or the **Full snapshot size** field in the console. The formula is as follows: ``` rate for Region x snapshot data size x volume initialization rate ``` You are billed the full amount as soon as the volume enters the `active` state. Failed requests are not billed. If you delete a volume before the volume initialization completes, you are still billed for the requested volume initialization rate. ## Use a snapshot that is enabled for fast snapshot restore If you create a volume from a snapshot that is enabled for fast snapshot restore, the volume is fully initialized at creation and it immediately delivers its full performance. For more information about using fast snapshot restore, see [Amazon EBS fast snapshot restore](ebs-fast-snapshot-restore.md). ## Manually initialize the volumes after creation You can manually initialize an Amazon EBS volume after creation to help minimize the performance impacts of volume initialization. You can use the following procedures to manually initialize an Amazon EBS volume after creation. **Important** While initializing Provisioned IOPS SSD volumes that were created from snapshots, the performance of the volume may drop below 50 percent of its expected level, which causes the volume to display a `warning` state in the **I/O Performance** status check. This is expected, and you can ignore the `warning` state on Provisioned IOPS SSD volumes while you are initializing them. For more information, see [Amazon EBS volume status checks](monitoring-volume-checks.md). ### Linux instances **To initialize a volume created from a snapshot on Linux** 1. Attach the newly-restored volume to your Linux instance. 1. Use the **lsblk** command to list the block devices on your instance. ``` $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvdf 202:80 0 30G 0 disk xvda1 202:1 0 8G 0 disk / ``` Here you can see that the new volume, `/dev/xvdf`, is attached, but not mounted (because there is no path listed under the `MOUNTPOINT` column). 1. Use the **dd** or **fio** utilities to read all of the blocks on the device. The **dd** command is installed by default on Linux systems, but **fio** is considerably faster because it allows multi-threaded reads. **Note** This step may take several minutes up to several hours, depending on your EC2 instance bandwidth, the IOPS provisioned for the volume, and the size of the volume. [**dd**] The `if` (input file) parameter should be set to the drive you wish to initialize. The `of` (output file) parameter should be set to the Linux null virtual device, `/dev/null`. The `bs` parameter sets the block size of the read operation; for optimal performance, this should be set to 1 MB. **Important** Incorrect use of **dd** can easily destroy a volume's data. Be sure to follow precisely the example command below. Only the `if=/dev/xvdf` parameter will vary depending on the name of the device you are reading. ``` $ sudo dd if=/dev/xvdf of=/dev/null bs=1M ``` [**fio**] If you have **fio** installed on your system, use the following command to initialize your volume. The `--filename` (input file) parameter should be set to the drive you wish to initialize. ``` $ sudo fio --filename=/dev/xvdf --rw=read --bs=1M --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize ``` To install **fio** on Amazon Linux, use the following command: ``` sudo yum install -y fio ``` To install **fio** on Ubuntu, use the following command: ``` sudo apt-get install -y fio ``` When the operation is finished, you will see a report of the read operation. Your volume is now ready for use. For more information, see [Make an Amazon EBS volume available for use](ebs-using-volumes.md). ### Windows instances Before using either tool, gather information about the disks on your system as follows: **To gather information about the system disks** 1. Use the **wmic** command to list the available disks on your system: ``` wmic diskdrive get size,deviceid ``` The following is example output: ``` DeviceID Size \\.\PHYSICALDRIVE2 80517265920 \\.\PHYSICALDRIVE1 80517265920 \\.\PHYSICALDRIVE0 128849011200 \\.\PHYSICALDRIVE3 107372805120 ``` 1. Identify the disk to initialize using **dd** or **fio**. The `C:` drive is on `\\.\PHYSICALDRIVE0`. You can use the `diskmgmt.msc` utility to compare drive letters to disk drive numbers if you are not sure which drive number to use. ------ #### [ Use the dd utility ] Complete the following procedures to install and use **dd** to initialize a volume. **Important considerations** + Initializing a volume takes from several minutes up to several hours, depending on your EC2 instance bandwidth, the IOPS provisioned for the volume, and the size of the volume. + Incorrect use of **dd** can easily destroy a volume's data. Be sure to follow this procedure precisely. **To install dd for Windows** The **dd** for Windows program provides a similar experience to the **dd** program that is commonly available for Linux and Unix systems, and it enables you to initialize Amazon EBS volumes that have been created from snapshots. The most recent beta versions support the `/dev/null` virtual device. If you install an earlier version, you can use the `nul` virtual device instead. Full documentation is available at [http://www.chrysocome.net/dd](http://www.chrysocome.net/dd). 1. Download the most recent binary version of **dd** for Windows from [http://www.chrysocome.net/dd](http://www.chrysocome.net/dd). 1. (Optional) Create a folder for command line utilities that is easy to locate and remember, such as `C:\bin`. If you already have a designated folder for command line utilities, you can use that folder instead in the following step. 1. Unzip the binary package and copy the `dd.exe` file to your command line utilities folder (for example, `C:\bin`). 1. Add the command line utilities folder to your Path environment variable so you can run the programs in that folder from anywhere. 1. Choose **Start**, open the context (right-click) menu for **Computer**, and then choose **Properties**. 1. Choose **Advanced system settings**, **Environment Variables**. 1. For **System Variables**, select the variable **Path** and choose **Edit**. 1. For **Variable value**, append a semicolon and the location of your command line utility folder (**;C:\$1bin\$1)** to the end of the existing value. 1. Choose **OK** to close the **Edit System Variable ** window. 1. Open a new command prompt window. The previous step doesn't update the environment variables in your current command prompt windows. The command prompt windows that you open now that you completed the previous step are updated. **To initialize a volume using dd for Windows** Run the following command to read all blocks on the specified device (and send the output to the `/dev/null` virtual device). This command safely initializes your existing data. ``` dd if=\\.\PHYSICALDRIVEn of=/dev/null bs=1M --progress --size ``` You might get an error if **dd** attempts to read beyond the end of the volume. You can safely ignore this error. If you used an earlier version of the **dd** command, it does not support the `/dev/null` device. Instead, you can use the `nul` device as follows. ``` dd if=\\.\PHYSICALDRIVEn of=nul bs=1M --progress --size ``` ------ #### [ Use the fio utility ] Complete the following procedures to install and use **fio** to initialize a volume. **To install **fio** for Windows** The **fio** for Windows program provides a similar experience to the **fio** program that is commonly available for Linux and Unix systems, and it allows you to initialize Amazon EBS volumes created from snapshots. For more information, see [https://github.com/axboe/fio](https://github.com/axboe/fio). 1. Download the [**fio** MSI](https://github.com/axboe/fio/releases) installer by expanding **Assets** for the latest release and selecting the MSI installer. 1. Install **fio**. **To initialize a volume using **fio** for Windows** 1. Run a command similar to the following to initialize a volume: ``` fio --filename=\\.\PHYSICALDRIVEn --rw=read --bs=1M --iodepth=32 --direct=1 --name=volume-initialize ``` 1. When the operation completes, you are ready to use your new volume. For more information, see [Make an Amazon EBS volume available for use](ebs-using-volumes.md). ------ # Monitor the status of Amazon EBS volume initialization When you create a volume, either from a snapshot or from another volume (volume copy), you can monitor the status of the volume initialization to determine whether the initialization process is complete. You can monitor volume initialization using the following options: **Topics** + [ ## AWS CLI and Amazon EC2 console ](#ebs-initialize-monitor-ec2) + [ ## Amazon EventBridge ](#ebs-initialize-monitor-ev) ## AWS CLI and Amazon EC2 console You can use the AWS CLI and Amazon EC2 console to check the status of the volume initialization at any time after the volume has been created. The following information is provided: + **Initialization type** (AWS CLI only) — Indicates the type of volume initialization used. `default` for fast snapshot restore and default volume initialization, `provisioned-rate` for Amazon EBS Provisioned Rate for Volume Initialization, and `volume-copy` for volume copy initialization. + **Estimated time to completion** (AWS CLI only) — Only for volumes created using a Amazon EBS Provisioned Rate for Volume Initialization. The estimated remaining time, in seconds, for the volume initialization to complete. + **Progress** — The progress, as a percentage (0-100), for the volume initialization process. For volumes initialized with fast snapshot restore, the progress moves to 100 percent immediately after creation. + **Initialization state** — The overall state of the volume initialization (`initializing` or `completed`). For volumes initialized with fast snapshot restore, the state moves to `completed` immediately after creation. **Note** It can take up to 5 minutes for the volume initialization information to be updated. ------ #### [ Console ] **To monitor status of volume initialization** 1. Open the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/). 1. In the navigation pane, choose **Volumes**. 1. Select the volume for which to check the volume initialization status. 1. The **Initialization state** field in the grid and **Details** tab provide progress information in the following format: *Initialization state (progress percentage)*. For example, *Initializing (75%)*. The possible initialization states include: *Initializing* and *Completed*. ------ #### [ AWS CLI ] **To monitor status of volume initialization** Use the [ describe-volume-status](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-volume-status.html) AWS CLI command to view the initialization status. `EstimatedTimeToCompleteInSeconds` is returned only for volumes created with an Amazon EBS Provisioned Rate for Volume Initialization. For example, the following command checks the initialization status for volume `vol-11111111111111111`, which was created with an Amazon EBS Provisioned Rate for Volume Initialization. ``` aws ec2 describe-volume-status --volume-id vol-01111111111111111 ``` The following is example output. ``` { "VolumeStatuses": [ { "Actions": [], "AvailabilityZone": "us-east-1a", "Events": [], "VolumeID": "vol-11111111111111111", "VolumeStatus": { "Details": [ { "Name": "io-enabled", "Status": "passed" }, { "Name": "io-performance", "Status": "not-applicable" }, { "Name": "initialization-state", "Status": "completed" } ], "Status": "ok" }, "InitializationStatusDetails": { "InitializationType": "provisioned-rate", "Progress": 75, "EstimatedTimeToCompleteInSeconds": 850 } } ] } ``` ------ ## Amazon EventBridge An Amazon EventBridge event is sent to your account within five minutes **after** the volume initialization has completed. You can create rules that trigger programmatic actions in response to these events. **Note** Events are emitted on a best effort basis. If you delete the volume before initialization completes, or within 5 minutes after initialization completes, you might not receive the event. For more information about the event, see [EBS volume initialization event](ebs-cloud-watch-events.md#volume-initialization-events). **To monitor status of volume initialization using EventBridge** 1. Open the Amazon EventBridge console at [https://console.aws.amazon.com/events/](https://console.aws.amazon.com/events/). 1. Choose **Rules**, **Create rule**. 1. For **Step 1**, do the following: 1. Specify a name and description for the rule. 1. For **Event bus**, choose the bus to receive the events. If you haven't created a custom event bus, keep **default**, or see [Creating an event bus](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-event-bus.html). 1. For **Rule type**, keep **Rule with an event pattern**. 1. Choose **Next**. 1. For **Step 2**, do the following: 1. For **Event source**, keep **AWS events or EventBridge partner events**. 1. For **Creation method**, choose **Custom pattern (JSON editor)**. 1. For **Event pattern**, add the following: ``` { "detail-type": ["EBS Volume Notification"], "source": ["aws.ec2"], "detail": { "event": ["initializeVolume"], "result": ["succeeded"] } } ``` For an example event, see [EBS volume initialization event](ebs-cloud-watch-events.md#volume-initialization-events). 1. Choose **Next**. 1. For **Step 3**, do the following: 1. For **Target types**, choose **AWS service**. 1. For **Select target**, choose **SNS topic**, and for **Topic** select the required topic. If you haven't created any topics, see [Creating a topic](https://docs.aws.amazon.com/sns/latest/dg/sns-create-topic.html). 1. For **Permissions**, keep **Use execution role (recommended)** selected. 1. For **Execution role**, keep **Create a new role for this specific resource** selected and the defauly role name. 1. Choose **Next**. 1. For **Step 4**, specify tags for the rule if needed, and then choose **Next**. 1. For **Step 5**, review the rule and then choose **Create rule**. # Configurable instance bandwidth weighting Instance bandwidth configuration (IBC) is a feature that enables you to adjust the allocation of network bandwidth between Amazon EBS and VPC networking for an Amazon EC2 instance. This feature can help you optimize performance for workloads with specific bandwidth requirements. Instance bandwidth configuration is supported on some instances only. For more information, see [Instance bandwidth weighting configuration](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-bandwidth-weighting.html#config-bw-support). For EBS performance, using the `ebs-1` bandwidth weighting increases the baseline EBS bandwidth by 25 percent while reducing the VPC networking bandwidth by the same absolute amount. This can be beneficial for I/O-intensive workloads that require higher EBS throughput. When planning your workload, carefully consider your I/O size and patterns. Smaller I/O sizes are generally less affected by bandwidth limitations, while larger I/O sizes or sequential workloads can experience more significant impacts from bandwidth changes. It's crucial to thoroughly test your specific workload to ensure optimal performance with your chosen bandwidth weighting. **Considerations** + Configurable instance bandwidth is supported on select instance types. For more information, see [ Supported instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-bandwidth-weighting.html#config-bw-support). + Using the `ebs-1` bandwidth weighting increases EBS bandwidth up to 25 percent, which can improve the performance of I/O-intensive applications. However, keep in mind that VPC networking bandwidth will be reduced by the same absolute amount (the combined bandwidth specification between EBS and networking does not change). + Changes in bandwidth weighting can significantly affect I/O performance. With the `vpc-1` bandwidth weighting, network bandwidth is increased, but you might experience lower than expected IOPS for EBS volumes. This is because you might reach the EBS bandwidth limit before the IOPS limit, especially with larger I/O sizes. For example, an instance type that typically supports 240,000 IOPS with 16 KiB I/O size might achieve fewer IOPS when using `vpc-1` bandwidth weight due to the decreased EBS bandwidth. + Always test your specific workload to ensure that your chosen bandwidth weighting meets your performance needs. + You can configure the bandwidth weighting during instance launch or modify it for stopped instances. For more information see [Configure bandwidth weighting for your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-bandwidth-weighting.html#config-bw-how-to). + You can configure instance bandwidth weighting at no additional costs. # Amazon EBS I/O characteristics and monitoring On a given volume configuration, certain I/O characteristics drive the performance behavior for your EBS volumes. + SSD-backed volumes, General Purpose SSD (`gp2` and `gp3`) and Provisioned IOPS SSD (`io1` and `io2`), deliver consistent performance whether an I/O operation is random or sequential. + HDD-backed volumes, Throughput Optimized HDD (`st1`) and Cold HDD (`sc1`), deliver optimal performance only when I/O operations are large and sequential. To understand how SSD and HDD volumes will perform in your application, it is important to know the connection between demand on the volume, the quantity of IOPS available to it, the time it takes for an I/O operation to complete, and the volume's throughput limits. **Topics** + [ ## IOPS ](#ebs-io-iops) + [ ## Volume queue length and latency ](#ebs-io-volume-queue) + [ ## I/O size and volume throughput limits ](#ebs-io-size-throughput-limits) + [ ## Monitor I/O characteristics using CloudWatch ](#ebs-io-metrics) + [ ## Monitor real-time I/O performance statistics ](#monitor-io-nvme) + [ ## Related resources ](#ebs-io-resources) ## IOPS IOPS are a unit of measure representing input/output operations per second. The operations are measured in KiB, and the underlying drive technology determines the maximum amount of data that a volume type counts as a single I/O. I/O size is capped at 256 KiB for SSD volumes and 1,024 KiB for HDD volumes because SSD volumes handle small or random I/O much more efficiently than HDD volumes. When small I/O operations are physically sequential, Amazon EBS attempts to merge them into a single I/O operation up to the maximum I/O size. Similarly, when I/O operations are larger than the maximum I/O size, Amazon EBS attempts to split them into smaller I/O operations. The following table shows some examples. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/ebs/latest/userguide/ebs-io-characteristics.html) Consequently, when you create an SSD-backed volume supporting 3,000 IOPS (either by provisioning an `io1` or `io2` volume with 3,000 IOPS, by sizing a `gp2` volume at 1,000 GiB, or by using a `gp3` volume), and you attach it to an EBS-optimized instance that can provide sufficient bandwidth, you can transfer up to 3,000 I/Os of data per second, with throughput determined by I/O size. ## Volume queue length and latency The volume queue length is the number of pending I/O requests for a device. Latency is the true end-to-end client time of an I/O operation, in other words, the time elapsed between sending an I/O to EBS and receiving an acknowledgement from EBS that the I/O read or write is complete. Queue length must be correctly calibrated with I/O size and latency to avoid creating bottlenecks either on the guest operating system or on the network link to EBS. Optimal queue length varies for each workload, depending on your particular application's sensitivity to IOPS and latency. If your workload is not delivering enough I/O requests to fully use the performance available to your EBS volume, then your volume might not deliver the IOPS or throughput that you have provisioned. Transaction-intensive applications are sensitive to increased I/O latency and are well-suited for SSD-backed volumes. You can maintain high IOPS while keeping latency down by maintaining a low queue length and a high number of IOPS available to the volume. Consistently driving more IOPS to a volume than it has available can cause increased I/O latency. For maximum consistency, a volume must maintain an average queue depth (rounded to the nearest whole number) of one for every 1,000 provisioned IOPS in a minute. For example, for a volume provisioned with 3,000 IOPS, the queue depth average must be 3. Throughput-intensive applications are less sensitive to increased I/O latency, and are well-suited for HDD-backed volumes. You can maintain high throughput to HDD-backed volumes by maintaining a high queue length when performing large, sequential I/O. ## I/O size and volume throughput limits For SSD-backed volumes, if your I/O size is very large, you may experience a smaller number of IOPS than you provisioned because you are hitting the throughput limit of the volume. For example, a `gp2` volume under 1,000 GiB with burst credits available has an IOPS limit of 3,000 and a volume throughput limit of 250 MiB/s. If you are using a 256 KiB I/O size, your volume reaches its throughput limit at 1000 IOPS (1000 x 256 KiB = 250 MiB). For smaller I/O sizes (such as 16 KiB), this same volume can sustain 3,000 IOPS because the throughput is well below 250 MiB/s. (These examples assume that your volume's I/O is not hitting the throughput limits of the instance.) For more information about the throughput limits for each EBS volume type, see [Amazon EBS volume types](ebs-volume-types.md). For smaller I/O operations, you may see a higher-than-provisioned IOPS value as measured from inside your instance. This happens when the instance operating system merges small I/O operations into a larger operation before passing them to Amazon EBS. If your workload uses sequential I/Os on HDD-backed `st1` and `sc1` volumes, you may experience a higher than expected number of IOPS as measured from inside your instance. This happens when the instance operating system merges sequential I/Os and counts them in 1,024 KiB-sized units. If your workload uses small or random I/Os, you may experience a lower throughput than you expect. This is because we count each random, non-sequential I/O toward the total IOPS count, which can cause you to hit the volume's IOPS limit sooner than expected. Whatever your EBS volume type, if you are not experiencing the IOPS or throughput you expect in your configuration, ensure that your EC2 instance bandwidth is not the limiting factor. You should always use a current-generation, EBS-optimized instance (or one that includes 10 Gb/s network connectivity) for optimal performance. Another possible cause for not experiencing the expected IOPS is that you are not driving enough I/O to the EBS volumes. ## Monitor I/O characteristics using CloudWatch You can monitor these I/O characteristics with each volume's [CloudWatch volume metrics](using_cloudwatch_ebs.md#ebs-volume-metrics). **Monitor for stalled I/O** `VolumeStalledIOCheck` monitors the status of your EBS volumes to determine when your volumes are impaired. The metric is a binary value that will return a `0` (pass) or a `1` (fail) status based on whether or not the EBS volume can complete I/O operations. If the `VolumeStalledIOCheck` metric fails, you can either wait for AWS to resolve the issue, or you can take actions, such as replacing the affected volume or stopping and restarting the instance to which the volume is attached. In most cases, when this metric fails, EBS will automatically diagnose and recover your volume within a few minutes. You can use the [Pause I/O](ebs-fis.md) action in AWS Fault Injection Service to run controlled experiments to test your architecture and monitoring based on this metric to improve your resiliency to storage faults. **Monitor I/O latency for a volume** You can monitor the average latency for read and write operations for an Amazon EBS volume using the `VolumeAvgReadLatency` and `VolumeAvgWriteLatency` metrics respectively. You can use the [Latency Injection](ebs-fis-latency-injection.md) action in AWS Fault Injection Service to run controlled experiments to test your architecture and monitoring based on this metric to improve your resiliency to storage performance degradation. If your I/O latency is higher than you require, make sure that your application is not attempting to drive more IOPS or throughput than you have provisioned for your volume. You can use the `VolumeAvgIOPS` and `VolumeAvgThroughput` metrics to monitor the average IOPS and throughput driven to your volume in a minute and then compare that with the volume's provisioned IOPS and throughput. If the volume does not drive any operations during the minute, the metrics will report a value of zero (`0`). If bursts of high IOPS or throughput occurred for a shorter time than the minute interval then the volume experiences micro-bursting, but the average IOPS and throughput metrics may report that you are driving lower performance than your volume's provisioned IOPS or througput limits. To identify whether your volume experiences performance bursts in a given minute, you can use the `VolumeIOPSExceededCheck` and `VolumeThroughputExceededCheck` metrics. You can monitor these metrics to determine whether your workload consistently attempted to drive IOPS or throughput that is greater than your volume's provisioned performance in a given minute. If the driven IOPS for any second within the minute consistently exceeds your volume's provisioned IOPS performance, the `VolumeIOPSExceededCheck` metric returns `1`. If the driven throughput for any second within the minute consistently exceeds your volume's provisioned throughput performance, the `VolumeThroughputExceededCheck` metric returns `1`. If driven IOPS and throughput is within your volume's provisioned performance, the metrics return `0`. If your application requires a greater number of IOPS than your volume can provide, you should consider using one of the following: + A `gp3`, `io2`, or `io1` volume that is provisioned with enough IOPS to achieve the required latency + A larger `gp2` volume that provides enough baseline IOPS performance HDD-backed `st1` and `sc1` volumes are designed to perform best with workloads that take advantage of the 1,024 KiB maximum I/O size. To determine your volume's average I/O size, divide `VolumeWriteBytes` by `VolumeWriteOps`. The same calculation applies to read operations. If average I/O size is below 64 KiB, increasing the size of the I/O operations sent to an `st1` or `sc1` volume should improve performance. **Monitor burst bucket balance for `gp2`, `st1`, and `sc1` volumes** `BurstBalance` displays the burst bucket balance for `gp2`, `st1`, and `sc1` volumes as a percentage of the remaining balance. When your burst bucket is depleted, volume I/O (for `gp2` volumes) or volume throughput (for `st1` and `sc1` volumes) is throttled to the baseline. Check the `BurstBalance` value to determine whether your volume is being throttled for this reason. For a complete list of the available Amazon EBS metrics, see [Amazon CloudWatch metrics for Amazon EBS](using_cloudwatch_ebs.md) and [ Amazon EBS metrics for Nitro-based instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html#ebs-metrics-nitro). ## Monitor real-time I/O performance statistics You can access real-time detailed performance statistics for Amazon EBS volumes that are attached to Nitro-based Amazon EC2 instances. You can combine these statistics to derive average latency and IOPS, or to check whether I/O operations are completing. You can also view the total amount of time that your application has exceeded your EBS volume's or the attached instance's provisioned IOPS or throughput limits. By tracking increases in these statistics over time, you can identify whether you need to increase your provisioned IOPS or throughput limits to optimize your application's performance. The detailed performance statistics also include histograms for read and write I/O operations, which provide a distribution of your I/O latency by keeping track of the total number of I/O operations completed within a latency band. For more information, see [Amazon EBS detailed performance statistics](nvme-detailed-performance-stats.md). ## Related resources For more information about Amazon EBS I/O characteristics, see the following re:Invent presentation: [Amazon EBS: Designing for Performance](https://www.youtube.com/watch?v=2wKgha8CZ_w). # Amazon EBS and RAID configuration With Amazon EBS, you can use any of the standard RAID configurations that you can use with a traditional bare metal server, as long as that particular RAID configuration is supported by the operating system for your instance. This is because all RAID is accomplished at the software level. Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. This replication makes Amazon EBS volumes ten times more reliable than typical commodity disk drives. For more information, see [Amazon EBS features](https://aws.amazon.com/ebs/features/). **Topics** + [ ## RAID configuration options ](#raid-config-options) + [ ## Create a RAID 0 array ](#create-raid-array) + [ ## Create snapshots of volumes in a RAID array ](#ebs-snapshots-raid-array) ## RAID configuration options Creating a RAID 0 array allows you to achieve a higher level of performance for a file system than you can provision on a single Amazon EBS volume. Use RAID 0 when I/O performance is of the utmost importance. With RAID 0, I/O is distributed across the volumes in a stripe. If you add a volume, you get the straight addition of throughput and IOPS. However, keep in mind that performance of the stripe is limited to the worst performing volume in the set, and that the loss of a single volume in the set results in a complete data loss for the array. The resulting size of a RAID 0 array is the sum of the sizes of the volumes within it, and the bandwidth is the sum of the available bandwidth of the volumes within it. For example, two 500 GiB `io1` volumes with 4,000 provisioned IOPS each create a 1,000 GiB RAID 0 array with an available bandwidth of 8,000 IOPS and 1,000 MiB/s of throughput. **Important** RAID 5 and RAID 6 are not recommended for Amazon EBS because the parity write operations of these RAID modes consume some of the IOPS available to your volumes. Depending on the configuration of your RAID array, these RAID modes provide 20-30% fewer usable IOPS than a RAID 0 configuration. Increased cost is a factor with these RAID modes as well; when using identical volume sizes and speeds, a 2-volume RAID 0 array can outperform a 4-volume RAID 6 array that costs twice as much. RAID 1 is also not recommended for use with Amazon EBS. RAID 1 requires more Amazon EC2 to Amazon EBS bandwidth than non-RAID configurations because the data is written to multiple volumes simultaneously. In addition, RAID 1 does not provide any write performance improvement. ## Create a RAID 0 array Use the following procedure to create the RAID 0 array. **Considerations** + Before you perform this procedure, you must decide how large your RAID 0 array should be and how many IOPS to provision. + Create volumes with identical size and IOPS performance values for your array. Make sure you do not create an array that exceeds the available bandwidth of your EC2 instance. + You should avoid booting from a RAID volume. If one of the devices fails, you might be unable to boot the operating system. ### Linux instances **To create a RAID 0 array on Linux** 1. Create the Amazon EBS volumes for your array. For more information, see [Create an Amazon EBS volume](ebs-creating-volume.md). 1. Attach the Amazon EBS volumes to the instance that you want to host the array. For more information, see [Attach an Amazon EBS volume to an Amazon EC2 instance](ebs-attaching-volume.md). 1. Use the **mdadm** command to create a logical RAID device from the newly attached Amazon EBS volumes. Substitute the number of volumes in your array for *number\$1of\$1volumes* and the device names for each volume in the array (such as `/dev/xvdf`) for *device\$1name*. You can also substitute *MY\$1RAID* with your own unique name for the array. **Note** You can list the devices on your instance with the **lsblk** command to find the device names. To create a RAID 0 array, run the following command (note the `--level=0` option to stripe the array): ``` [ec2-user ~]$ sudo mdadm --create --verbose /dev/md0 --level=0 --name=MY_RAID --raid-devices=number_of_volumes device_name1 device_name2 ``` **Tip** If you get the `mdadm: command not found` error, use the following command to install mdadm: `sudo yum install mdadm`. 1. Allow time for the RAID array to initialize and synchronize. You can track the progress of these operations with the following command: ``` [ec2-user ~]$ sudo cat /proc/mdstat ``` The following is example output: ``` Personalities : [raid0] md0 : active raid0 xvdc[1] xvdb[0] 41910272 blocks super 1.2 512k chunks unused devices: ``` In general, you can display detailed information about your RAID array with the following command: ``` [ec2-user ~]$ sudo mdadm --detail /dev/md0 ``` The following is example output: ``` /dev/md0: Version : 1.2 Creation Time : Wed May 19 11:12:56 2021 Raid Level : raid0 Array Size : 41910272 (39.97 GiB 42.92 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Wed May 19 11:12:56 2021 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K Consistency Policy : none Name : MY_RAID UUID : 646aa723:db31bbc7:13c43daf:d5c51e0c Events : 0 Number Major Minor RaidDevice State 0 202 16 0 active sync /dev/sdb 1 202 32 1 active sync /dev/sdc ``` 1. Create a file system on your RAID array, and give that file system a label to use when you mount it later. For example, to create an ext4 file system with the label *MY\$1RAID*, run the following command: ``` [ec2-user ~]$ sudo mkfs.ext4 -L MY_RAID /dev/md0 ``` Depending on the requirements of your application or the limitations of your operating system, you can use a different file system type, such as ext3 or XFS (consult your file system documentation for the corresponding file system creation command). 1. To ensure that the RAID array is reassembled automatically on boot, create a configuration file to contain the RAID information: ``` [ec2-user ~]$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf ``` **Note** If you are using a Linux distribution other than Amazon Linux, you might need to modify this command. For example, you might need to place the file in a different location, or you might need to add the `--examine` parameter. For more information, run **man mdadm.conf** on your Linux instance. 1. Create a new ramdisk image to properly preload the block device modules for your new RAID configuration: ``` [ec2-user ~]$ sudo dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r) ``` 1. Create a mount point for your RAID array. ``` [ec2-user ~]$ sudo mkdir -p /mnt/raid ``` 1. Finally, mount the RAID device on the mount point that you created: ``` [ec2-user ~]$ sudo mount LABEL=MY_RAID /mnt/raid ``` Your RAID device is now ready for use. 1. (Optional) To mount this Amazon EBS volume on every system reboot, add an entry for the device to the `/etc/fstab` file. 1. Create a backup of your `/etc/fstab` file that you can use if you accidentally destroy or delete this file while you are editing it. ``` [ec2-user ~]$ sudo cp /etc/fstab /etc/fstab.orig ``` 1. Open the `/etc/fstab` file using your favorite text editor, such as **nano** or **vim**. 1. Comment out any lines starting with "`UUID=`" and, at the end of the file, add a new line for your RAID volume using the following format: ``` device_label mount_point file_system_type fs_mntops fs_freq fs_passno ``` The last three fields on this line are the file system mount options, the dump frequency of the file system, and the order of file system checks done at boot time. If you don't know what these values should be, then use the values in the example below for them (`defaults,nofail 0 2)`. For more information about `/etc/fstab` entries, see the **fstab** manual page (by entering **man fstab** on the command line). For example, to mount the ext4 file system on the device with the label MY\$1RAID at the mount point `/mnt/raid`, add the following entry to `/etc/fstab`. **Note** If you ever intend to boot your instance without this volume attached (for example, so this volume could move back and forth between different instances), you should add the `nofail` mount option that allows the instance to boot even if there are errors in mounting the volume. Debian derivatives, such as Ubuntu, must also add the `nobootwait` mount option. ``` LABEL=MY_RAID /mnt/raid ext4 defaults,nofail 0 2 ``` 1. After you've added the new entry to `/etc/fstab`, you need to check that your entry works. Run the **sudo mount -a** command to mount all file systems in `/etc/fstab`. ``` [ec2-user ~]$ sudo mount -a ``` If the previous command does not produce an error, then your `/etc/fstab` file is OK and your file system will mount automatically at the next boot. If the command does produce any errors, examine the errors and try to correct your `/etc/fstab`. **Warning** Errors in the `/etc/fstab` file can render a system unbootable. Do not shut down a system that has errors in the `/etc/fstab` file. 1. (Optional) If you are unsure how to correct `/etc/fstab` errors, you can always restore your backup `/etc/fstab` file with the following command. ``` [ec2-user ~]$ sudo mv /etc/fstab.orig /etc/fstab ``` ### Windows instances **To create a RAID 0 array on Windows** 1. Create the Amazon EBS volumes for your array. For more information, see [Create an Amazon EBS volume](ebs-creating-volume.md). 1. Attach the Amazon EBS volumes to the instance that you want to host the array. For more information, see [Attach an Amazon EBS volume to an Amazon EC2 instance](ebs-attaching-volume.md). 1. Connect to your Windows instance. For more information, see [Connect to your Windows instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connecting_to_windows_instance.html). 1. Open a command prompt and type the **diskpart** command. ``` diskpart Microsoft DiskPart version 6.1.7601 Copyright (C) 1999-2008 Microsoft Corporation. On computer: WIN-BM6QPPL51CO ``` 1. At the `DISKPART` prompt, list the available disks with the following command. ``` DISKPART> list disk Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 30 GB 0 B Disk 1 Online 8 GB 0 B Disk 2 Online 8 GB 0 B ``` Identify the disks you want to use in your array and take note of their disk numbers. 1. Each disk you want to use in your array must be an online dynamic disk that does not contain any existing volumes. Use the following steps to convert basic disks to dynamic disks and to delete any existing volumes. 1. Select a disk you want to use in your array with the following command, substituting *n* with your disk number. ``` DISKPART> select disk n Disk n is now the selected disk. ``` 1. If the selected disk is listed as `Offline`, bring it online by running the **online disk** command. 1. If the selected disk does not have an asterisk in the `Dyn` column in the previous **list disk** command output, you need to convert it to a dynamic disk. ``` DISKPART> convert dynamic ``` **Note** If you receive an error that the disk is write protected, you can clear the read-only flag with the **ATTRIBUTE DISK CLEAR READONLY** command and then try the dynamic disk conversion again. 1. Use the **detail disk** command to check for existing volumes on the selected disk. ``` DISKPART> detail disk XENSRC PVDISK SCSI Disk Device Disk ID: 2D8BF659 Type : SCSI Status : Online Path : 0 Target : 1 LUN ID : 0 Location Path : PCIROOT(0)#PCI(0300)#SCSI(P00T01L00) Current Read-only State : No Read-only : No Boot Disk : No Pagefile Disk : No Hibernation File Disk : No Crashdump Disk : No Clustered Disk : No Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 2 D NEW VOLUME FAT32 Simple 8189 MB Healthy ``` Note any volume numbers on the disk. In this example, the volume number is 2. If there are no volumes, you can skip the next step. 1. (Only required if volumes were identified in the previous step) Select and delete any existing volumes on the disk that you identified in the previous step. **Warning** This destroys any existing data on the volume. 1. Select the volume, substituting *n* with your volume number. ``` DISKPART> select volume n Volume n is the selected volume. ``` 1. Delete the volume. ``` DISKPART> delete volume DiskPart successfully deleted the volume. ``` 1. Repeat these substeps for each volume you need to delete on the selected disk. 1. Repeat [Step 6](#windows_raid_disk_step) for each disk you want to use in your array. 1. Verify that the disks you want to use are now dynamic. In this case, we're using disks 1 and 2 for the RAID volume. ``` DISKPART> list disk Disk ### Status Size Free Dyn Gpt -------- ------------- ------- ------- --- --- Disk 0 Online 30 GB 0 B Disk 1 Online 8 GB 0 B * Disk 2 Online 8 GB 0 B * ``` 1. Create your raid array. On Windows, a RAID 0 volume is referred to as a striped volume. To create a striped volume array on disks 1 and 2, use the following command (note the `stripe` option to stripe the array): ``` DISKPART> create volume stripe disk=1,2 DiskPart successfully created the volume. ``` 1. Verify your new volume. ``` DISKPART> list volume DISKPART> list volume Volume ### Ltr Label Fs Type Size Status Info ---------- --- ----------- ----- ---------- ------- --------- -------- Volume 0 C NTFS Partition 29 GB Healthy System Volume 1 RAW Stripe 15 GB Healthy ``` Note that the `Type` column now indicates that Volume 1 is a `stripe` volume. 1. Select and format your volume so that you can begin using it. 1. Select the volume you want to format, substituting *n* with your volume number. ``` DISKPART> select volume n Volume n is the selected volume. ``` 1. Format the volume. **Note** To perform a full format, omit the `quick` option. ``` DISKPART> format quick recommended label="My new volume" 100 percent completed DiskPart successfully formatted the volume. ``` 1. Assign an available drive letter to your volume. ``` DISKPART> assign letter f DiskPart successfully assigned the drive letter or mount point. ``` Your new volume is now ready to use. ## Create snapshots of volumes in a RAID array If you want to back up the data on the EBS volumes in a RAID array using snapshots, you must ensure that the snapshots are consistent. This is because the snapshots of these volumes are created independently. To restore EBS volumes in a RAID array from snapshots that are out of sync would degrade the integrity of the array. To create a consistent set of snapshots for your RAID array, use [EBS multi-volume snapshots](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateSnapshots.html). Multi-volume snapshots allow you to take point-in-time, data coordinated, and crash-consistent snapshots across multiple EBS volumes attached to an EC2 instance. You do not have to stop your instance to coordinate between volumes to ensure consistency because snapshots are automatically taken across multiple EBS volumes. For more information, see the steps for creating multi-volume snapshots under [Create Amazon EBS snapshots](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-creating-snapshot.html). # Benchmark Amazon EBS volumes You can test the performance of Amazon EBS volumes by simulating I/O workloads. The process is as follows: 1. Launch an EBS-optimized instance. 1. Create new EBS volumes. 1. Attach the volumes to your EBS-optimized instance. 1. Configure and mount the block device. 1. Install a tool to benchmark I/O performance. 1. Benchmark the I/O performance of your volumes. 1. Delete your volumes and terminate your instance so that you don't continue to incur charges. **Important** Some of the procedures result in the destruction of existing data on the EBS volumes you benchmark. The benchmarking procedures are intended for use on volumes specially created for testing purposes, not production volumes. ## Set up your instance To get optimal performance from EBS volumes, we recommend that you use an EBS-optimized instance. EBS-optimized instances deliver dedicated bandwidth between Amazon EC2 and Amazon EBS, with specifications depending on the instance type. To create an EBS-optimized instance, choose **Launch as an EBS-optimized instance** when launching the instance using the Amazon EC2 console, or specify **--ebs-optimized** when using the command line. Be sure that you select an instance type that supports this option. ### Set up Provisioned IOPS SSD or General Purpose SSD volumes To create Provisioned IOPS SSD (`io1` and `io2`) or General Purpose SSD (`gp2` and `gp3`) volumes using the Amazon EC2 console, for **Volume type**, choose **Provisioned IOPS SSD (io1)**, **Provisioned IOPS SSD (io2)**, **General Purpose SSD (gp2)**, or **General Purpose SSD (gp3)**. At the command line, specify `io1`, `io2`, `gp2`, or `gp3` for the **--volume-type** parameter. For `io1`, `io2`, and `gp3` volumes, specify the number of I/O operations per second (IOPS) for the **--iops** parameter. For more information, see [Amazon EBS volume types](ebs-volume-types.md) and [Create an Amazon EBS volume](ebs-creating-volume.md). (*Linux instances only*) For the example tests, we recommend that you create a RAID 0 array with 6 volumes, which offers a high level of performance. Because you are charged by gigabytes provisioned (and the number of provisioned IOPS for io1, io2, and gp3 volumes), not the number of volumes, there is no additional cost for creating multiple, smaller volumes and using them to create a stripe set. If you're using Oracle Orion to benchmark your volumes, it can simulate striping the same way that Oracle ASM does, so we recommend that you let Orion do the striping. If you are using a different benchmarking tool, you need to stripe the volumes yourself. For more information about how to create a RAID 0 array, see [Create a RAID 0 array](raid-config.md#create-raid-array). ### Set up Throughput Optimized HDD (`st1`) or Cold HDD (`sc1`) volumes To create an `st1` volume, choose **Throughput Optimized HDD** when creating the volume using the Amazon EC2 console, or specify **--type `st1`** when using the command line. To create an `sc1` volume, choose Cold HDD when creating the volume using the Amazon EC2 console, or specify **--type `sc1`** when using the command line. For information about creating EBS volumes, see [Create an Amazon EBS volume](ebs-creating-volume.md). For information about attaching these volumes to your instance, see [Attach an Amazon EBS volume to an Amazon EC2 instance](ebs-attaching-volume.md). (*Linux instances only*) AWS provides a JSON template for use with CloudFormation that simplifies this setup procedure. Access the [template](https://s3.amazonaws.com/cloudformation-examples/community/st1_cloudformation_template.json) and save it as a JSON file. CloudFormation allows you to configure your own SSH keys and offers an easier way to set up a performance test environment to evaluate `st1` volumes. The template creates a current-generation instance and a 2 TiB `st1` volume, and attaches the volume to the instance at `/dev/xvdf`. **(*Linux instances only*) To create an HDD volume using the template** 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Choose **Create Stack**. 1. Choose **Upload a Template to Amazon S3** and select the JSON template you previously obtained. 1. Give your stack a name like “ebs-perf-testing”, and select an instance type (the default is r3.8xlarge) and SSH key. 1. Choose **Next** twice, and then choose **Create Stack**. 1. After the status for your new stack moves from **CREATE\$1IN\$1PROGRESS** to **COMPLETE**, choose **Outputs** to get the public DNS entry for your new instance, which will have a 2 TiB `st1` volume attached to it. 1. Connect using SSH to your new stack as user **ec2-user**, with the hostname obtained from the DNS entry in the previous step. 1. Proceed to [Install benchmark tools](#install_tools). ## Install benchmark tools The following tables lists some of the possible tools you can use to benchmark the performance of EBS volumes. ### Linux instances | Tool | Description | | --- | --- | | fio | For benchmarking I/O performance. (Note that **fio** has a dependency on `libaio-devel`.) To install **fio** on Amazon Linux, run the following command: 
$ sudo yum install -y fio
To install **fio** on Ubuntu, run the following command: 
sudo apt-get install -y fio
| | [Oracle Orion Calibration Tool](https://docs.oracle.com/cd/E18283_01/server.112/e16638/iodesign.htm#BABFCFBC) | For calibrating the I/O performance of storage systems to be used with Oracle databases. | ### Windows instances | Tool | Description | | --- | --- | | [DiskSpd](https://github.com/microsoft/diskspd/releases) | DiskSpd is a storage performance tool from the Windows, Windows Server, and Cloud Server Infrastructure engineering teams at Microsoft. It is available for download at [https://github.com/Microsoft/diskspd/releases](https://github.com/Microsoft/diskspd/releases). After you download the `diskspd.exe` executable file, open a command prompt with administrative rights (by choosing "Run as Administrator"), and then navigate to the directory where you copied the `diskspd.exe` file. Copy the desired `diskspd.exe` executable file from the appropriate executable folder (`amd64fre`, `armfre` or `x86fre)` to a short, simple path like `C:\DiskSpd`. In most cases you will want the 64-bit version of DiskSpd from the `amd64fre` folder. The source code for DiskSpd is hosted on GitHub at: [https://github.com/Microsoft/diskspd](https://github.com/Microsoft/diskspd). | | CrystalDiskMark | CrystalDiskMark is a simple disk benchmark software. It is available for download at [https://crystalmark.info/en/software/crystaldiskmark/](https://crystalmark.info/en/software/crystaldiskmark/). | These benchmarking tools support a wide variety of test parameters. You should use commands that approximate the workloads your volumes will support. These commands provided below are intended as examples to help you get started. ## Choose the volume queue length Choosing the best volume queue length based on your workload and volume type. ### Queue length on SSD-backed volumes To determine the optimal queue length for your workload on SSD-backed volumes, we recommend that you target a queue length of 1 for every 1000 IOPS available (baseline for General Purpose SSD volumes and the provisioned amount for Provisioned IOPS SSD volumes). Then you can monitor your application performance and tune that value based on your application requirements. Increasing the queue length is beneficial until you achieve the provisioned IOPS, throughput or optimal system queue length value, which is currently set to 32. For example, a volume with 3,000 provisioned IOPS should target a queue length of 3. You should experiment with tuning these values up or down to see what performs best for your application. ### Queue length on HDD-backed volumes To determine the optimal queue length for your workload on HDD-backed volumes, we recommend that you target a queue length of at least 4 while performing 1MiB sequential I/Os. Then you can monitor your application performance and tune that value based on your application requirements. For example, a 2 TiB `st1` volume with burst throughput of 500 MiB/s and IOPS of 500 should target a queue length of 4, 8, or 16 while performing 1,024 KiB, 512 KiB, or 256 KiB sequential I/Os respectively. You should experiment with tuning these values up or down to see what performs best for your application. ## Disable C-states Before you run benchmarking, you should disable processor C-states. Temporarily idle cores in a supported CPU can enter a C-state to save power. When the core is called on to resume processing, a certain amount of time passes until the core is again fully operational. This latency can interfere with processor benchmarking routines. For more information about C-states and which EC2 instance types support them, see [Processor state control for your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor_state_control.html). ### Linux instances You can disable C-states on Amazon Linux, RHEL, and CentOS as follows: 1. Get the number of C-states. ``` $ cpupower idle-info | grep "Number of idle states:" ``` 1. Disable the C-states from c1 to cN. Ideally, the cores should be in state c0. ``` $ for i in `seq 1 $((N-1))`; do cpupower idle-set -d $i; done ``` ### Windows instances You can disable C-states on Windows as follows: 1. In PowerShell, get the current active power scheme. ``` $current_scheme = powercfg /getactivescheme ``` 1. Get the power scheme GUID. ``` (Get-WmiObject -class Win32_PowerPlan -Namespace "root\cimv2\power" -Filter "ElementName='High performance'").InstanceID ``` 1. Get the power setting GUID. ``` (Get-WmiObject -class Win32_PowerSetting -Namespace "root\cimv2\power" -Filter "ElementName='Processor idle disable'").InstanceID ``` 1. Get the power setting subgroup GUID. ``` (Get-WmiObject -class Win32_PowerSettingSubgroup -Namespace "root\cimv2\power" -Filter "ElementName='Processor power management'").InstanceID ``` 1. Disable C-states by setting the value of the index to 1. A value of 0 indicates that C-states are disabled. ``` powercfg /setacvalueindex 1 ``` 1. Set active scheme to ensure the settings are saved. ``` powercfg /setactive ``` ## Perform benchmarking The following procedures describe benchmarking commands for various EBS volume types. Run the following commands on an EBS-optimized instance with attached EBS volumes. If the EBS volumes were created from snapshots, be sure to initialize them before benchmarking. For more information, see [Manually initialize the volumes after creation](initalize-volume.md#ebs-initialize). **Tip** You can use the I/O latency histograms provided by the EBS detailed performance statistics to compare the distribution of I/O performance in your benchmarking tests. For more information, see [Amazon EBS detailed performance statistics](nvme-detailed-performance-stats.md). When you are finished testing your volumes, see the following topics for help cleaning up: [Delete an Amazon EBS volume](ebs-deleting-volume.md) and [Terminate your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html). ### Benchmark Provisioned IOPS SSD and General Purpose SSD volumes #### Linux instances Run **fio** on the RAID 0 array that you created. The following command performs 16 KB random write operations. ``` $ sudo fio --directory=/mnt/p_iops_vol0 --ioengine=psync --name fio_test_file --direct=1 --rw=randwrite --bs=16k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting --norandommap ``` The following command performs 16 KB random read operations. ``` $ sudo fio --directory=/mnt/p_iops_vol0 --name fio_test_file --direct=1 --rw=randread --bs=16k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting --norandommap ``` For more information about interpreting the results, see this tutorial: [Inspecting disk IO performance with fio](https://www.linux.com/training-tutorials/inspecting-disk-io-performance-fio/). #### Windows instances Run **DiskSpd** on the volume that you created. The following command will run a 30 second random I/O test using a 20GB test file located on the `C:` drive, with a 25% write and 75% read ratio, and an 8K block size. It will use eight worker threads, each with four outstanding I/Os, and a write entropy value seed of 1GB. The results of the test will be saved to a text file called `DiskSpeedResults.txt`. These parameters simulate a SQL Server OLTP workload. ``` diskspd -b8K -d30 -o4 -t8 -h -r -w25 -L -Z1G -c20G C:\iotest.dat > DiskSpeedResults.txt ``` For more information about interpreting the results, see this tutorial: [Inspecting disk IO performance with DiskSPd](https://sqlperformance.com/2015/08/io-subsystem/diskspd-test-storage). ### Benchmark `st1` and `sc1` volumes (Linux instances) Run **fio** on your `st1` or `sc1` volume. **Note** Prior to running these tests, set buffered I/O on your instance as described in [Increase read-ahead for high-throughput, read-heavy workloads on `st1` and `sc1` (*Linux instances only*)](ebs-performance.md#read_ahead). The following command performs 1 MiB sequential read operations against an attached `st1` block device (for example, `/dev/xvdf`): ``` $ sudo fio --filename=/dev/ --direct=1 --rw=read --randrepeat=0 --ioengine=libaio --bs=1024k --iodepth=8 --time_based=1 --runtime=180 --name=fio_direct_read_test ``` The following command performs 1 MiB sequential write operations against an attached `st1` block device: ``` $ sudo fio --filename=/dev/ --direct=1 --rw=write --randrepeat=0 --ioengine=libaio --bs=1024k --iodepth=8 --time_based=1 --runtime=180 --name=fio_direct_write_test ``` Some workloads perform a mix of sequential reads and sequential writes to different parts of the block device. To benchmark such a workload, we recommend that you use separate, simultaneous **fio** jobs for reads and writes, and use the **fio** `offset_increment` option to target different block device locations for each job. Running this workload is a bit more complicated than a sequential-write or sequential-read workload. Use a text editor to create a fio job file, called `fio_rw_mix.cfg` in this example, that contains the following: ``` [global] clocksource=clock_gettime randrepeat=0 runtime=180 [sequential-write] bs=1M ioengine=libaio direct=1 iodepth=8 filename=/dev/ do_verify=0 rw=write rwmixread=0 rwmixwrite=100 [sequential-read] bs=1M ioengine=libaio direct=1 iodepth=8 filename=/dev/ do_verify=0 rw=read rwmixread=100 rwmixwrite=0 offset=100g ``` Then run the following command: ``` $ sudo fio fio_rw_mix.cfg ``` For more information about interpreting the results, see this tutorial: [Inspecting disk I/O performance with fio](https://www.linux.com/training-tutorials/inspecting-disk-io-performance-fio/). Multiple **fio** jobs for direct I/O, even though using sequential read or write operations, can result in lower than expected throughput for `st1` and `sc1` volumes. We recommend that you use one direct I/O job and use the `iodepth` parameter to control the number of concurrent I/O operations.