Cost
You are responsible for the cost of the AWS services used while running this solution. As of this revision, the cost for running this solution with the default settings in the US East (N. Virginia) Region is approximately $3,096 for a 15-day migration with 100 TB of existing data and 15 MBps of live traffic. These costs are for the resources shown in the sample cost table.
We recommend creating
a budget through AWS Cost Explorer
A migration typically has different duration and data volume for
the individual steps described in the
OpenSearch Migrations Wiki
In the following example, we outline the cost of a 15-day migration with the following schedule:
Example 15-day migration schedule
To understand the costs, we need to map these steps into cost components, including data retention periods as applicable. This yields the following schedule:
Example 15-day migration schedule mapped to solution components
-
Core Services – 15 days
-
Capture Proxy – 6 days
-
Capture Proxy Data Retention Period – 4 days
-
Snapshot – 1 day
-
Snapshot Data Retention Period – 9 days
-
Reindex from Snapshot – 3 days
-
Traffic Replayer – 2 days
-
Traffic Replayer Data Retention Period – 4 days
-
Target Proxy – 4 days
Using this schedule, the following is an example of a customer performing a zero-downtime migration of a cluster with 100 TB of primary shard data with 15 MBps aggregated request-response throughput. The ongoing request throughput over the 6-day capture timeline gives a capture/replay volume of about 7.8 TB.
Assumptions:
-
No data is retained after the teardown step, representing no resources and cost after day 15.
-
Deployment in US East (N. Virginia).
-
The Source Cluster, Target Cluster, and VPC are imported and cost excluded from this calculation, including data transfer cost based on VPC setup. We recommend that the VPC includes an Amazon S3 Gateway Endpoint and cluster data communication over VPC Interface Endpoints to reduce data transfer cost.
This gives us the following cost table:
| AWS service | Dimensions | Cost [USD] |
|---|---|---|
| Core services | ||
| Amazon ECS |
Migration Management Console: Task vCPU: 1 task × 24 hours per day × 15 days × (0.50 vCPU × 0.04048 USD per hour) = $7.29 Task Memory: 1 task × 24 hours per day × 15 days × (1 GB × 0.004445 USD per hour) = $1.60 |
$8.89 |
| AWS Lambda |
< 10 requests × 0.20 per 1M requests = ~$0.00 < 10 seconds × $0.0000166667 per GB-second = ~$0.00 |
~$0.00 |
| AWS Secrets Manager |
($0.40 per secret-month × 10 secrets × 15 days) / 30 days per month = $2.00 |
$2.00 |
| Amazon Route 53 | ($0.50 per Hosted Zone × 15 days) / 30 days per month = $0.25 |
$0.25 |
| Amazon Elastic Container Registry (Amazon ECR) | (3 GB × $0.10 GB/month × 15 days) / 30 days per month = $0.15 |
$0.15 |
| Amazon EC2 |
Bootstrap Box (t3.large): 1 × $0.0832/hr × 24 hours per day × 15 days = $29.95 |
$29.95 |
| Amazon EBS |
Bootstrap Box: (50 GB × $0.08/GB-month × 15 days) / 30 days per month = $2.00 |
$2.00 |
| Amazon Virtual Private Cloud (Amazon VPC) |
Bootstrap Box VPC – NAT Gateway: 1 × 24 hours per day × 15 days × $0.045/hour = $16.20 30 GB × $0.045/GB = $1.35 |
$17.55 |
| Elastic Load Balancing |
Application Load Balancer hours: 15 days × 24 hours × $0.0225 per Application Load Balancer-hour = $8.10 |
$8.10 |
| Core services total: | $68.89 | |
| Miscellaneous services | ||
| Amazon CloudWatch |
NoteThis is a high estimate for aggregated charges over all components for a large migration. Log data ingested (200 GB): $0.50 per GB × 200 GB = $100.00 Archived log charges (assume log data compresses to 30 GB): ($0.03 per GB-month × 30 GB × 15 days) / 30 days per month = $0.45 Metrics (200 metrics): (200 metrics × $0.30 per metric-month × 15 days) / 30 days per month = $60.00 NoteMetrics are metered only while being sent. Actual usage and cost might be lower. |
$160.45 |
| AWS X-Ray |
1M traces × $5.00 per million traces = $5.00 |
$5.00 |
| Miscellaneous services total: | $165.45 | |
|
Optional services |
||
|
Capture proxy (7.776 TB), 6 days running; 4 days data retention) |
||
| Amazon Managed Streaming for Apache Kafka (Amazon MSK) |
NoteAssume deployment in 2 Availability Zones. Broker (6 days running + 4 days data retention): 2 nodes (M5.large) × $0.21 node per hour × 10 days × 24 hours per day = $151.20 Storage (9,331 GB Provisioned (20% buffer) for 6 days running + 4 days data retention): (2 nodes × 9,331 GB × $0.10 per GB-month × 10 days) / 30 days per month = $620.07 Data transfer: 7,776 GB × $0.01 per GB inbound = $77.76 |
$849.03 |
| Amazon ECS |
Task vCPU: 4 tasks × 0.5 vCPU × 0.04048 USD per vCPU-hour × 6 days × 24 hours per day = $11.66 Task memory: 4 tasks × 2 GB × 0.004445 USD per GB-hour × 6 days × 24 hours per day = $5.12 |
$16.78 |
| Elastic Load Balancing |
Load Balancer Capacity Unity (LCU) (assuming GB is the cost dimension): 7,776 GB × $0.008 per LCU = $62.21 |
$62.21 |
|
Capture proxy total: |
$928.02 |
|
|
Target proxy (15 MBps for 4 days (5.184 TB)) |
||
| Amazon ECS |
Task vCPU: 4 tasks × 0.5 vCPU × 0.04048 USD per vCPU-hour) × 4 days × 24 hours per day = $7.77 Task memory: 4 tasks × 2 GB × 0.004445 USD per GB-hour × 4 days × 24 hours per day = $3.41 |
$11.18 |
| Elastic Load Balancing |
LCU (assuming GB is the cost dimension): 5,184 GB × $0.008 per LCU = $41.47 |
$41.47 |
|
Target proxy total: |
$52.65 |
|
|
Replay (7.776 TB over 2 days, retention 4 days) |
||
| Amazon ECS |
Task vCPU: 1 task × 1 vCPU × 0.04048 USD per vCPU-hour × 2 days × 24 hours per day = $1.94 Task memory: 1 task × 4 GB × 0.004445 USD per GB-hour × 2 days × 24 hours per day = $0.86 |
$2.80 |
| Amazon MSK |
Intra-Region: 7,776 GB × 0.01 USD per GB outbound = $77.76 |
$77.76 |
| Amazon Elastic File System (Amazon EFS) |
NoteCalculating cost based on constant size of 15.552 TB (7.776 TB × 2). Actual cost might differ due to some data spending less than 5 days in EFS Infrequent Access or data inflation. EFS Standard (infrequent after 1 day): 15,552 GB × $0.30 per GB-month × 1 day / 30 days per month = $155.52 EFS Infrequent Access (5 days maximum): 15,552 GB × $0.025 per GB-month × 5 days / 30 days per month = $64.80 EFS Infrequent Access – Tiering: 15,552 GB × $0.01 per GB = $155.52 |
$375.84 |
|
Replay total: |
$456.40 |
|
|
Historical Backfill (100 TB Snapshot over 1 month) |
||
| Amazon ECS |
RFS: Each task is capable of delivering approximately 5 MBps of snapshot data. To deliver 100 TB over 3 days, 77 tasks are needed. Task vCPU: 77 tasks × 3 days × 24 hours per day × 2 vCPUs × $0.04048 per hour = $448.84 Task memory: 77 tasks × 3 days × 24 hours per day × 4 GB × $0.004445 per hour = $98.57 Task storage: 77 tasks × 3 days × 24 hours per day × (200GB - 20GB Free Tier) × $0.000111 per hour = $110.77 |
$657.34 |
| Amazon S3 |
NoteThis calculates the cost for all the data occurring in the S3 bucket for the entire duration. The actual cost might be less for storage due to partial rate while the snapshot is being taken. Snapshot storage: (100 TB × 1,000 GB per TB × $0.023 per GB-month × 10 days) / 30 days per month = $766.66 |
$766.66 |
|
Historical Backfill total: |
$1,424.00 |
|
|
Total for all components: |
$3,095.41 |
|
The Capture and Replay yields an effective cost per TB of $199.87, and the Historic Backfill is $15.41 per TB. For a given cluster, the amount of data in Historic Backfill is largely fixed, while the data for Capture and Replay is based on the time needed to capture. For this reason, it can be cost advantageous to scale up the target OpenSearch cluster beyond the final intended capacity for the Historic Backfill period to reduce the duration and quantity of data for Capture and Replay.
Taking the table above and dividing out the components cost, we can determine the following calculation method for cost. This is a broad estimation making assumptions on traffic pattern including size per request, and number of new connections. We can also subdivide this by migration type applicable.
| Component | Applicable migration type | Cost [USD] |
|---|---|---|
| Core services |
|
$4.59/day |
| Miscellaneous services |
|
$165.45 per large migration |
| Capture runtime |
|
$17.22/day + $8.00/TB-day + $18.00/TB |
| Capture data retention period |
|
$15.12/day + $8.00/TB-day |
| Snapshot |
|
$0.7667/TB-day |
| Snapshot data retention period |
|
$0.7667/TB-day |
| Reindex from snapshot |
|
$6.57/TB |
| Traffic Replayer |
|
$1.40/day + $50.00/TB + $1.6667/TB-day NoteDue to Amazon EFS-Intelligent Tiering, $1.6667/TB-day is not paid for the first day. |
| Traffic Replayer data retention period |
|
$1.6667/TB-day |
| Target proxy |
|
$2.10/day + $8.00/TB |
We can also identify the cost for full migrations of different sizes that follow the 15-day migration timeline. By applying the durations shown previously, as well as a conversion from TB/day to MB/s, we get the following formula for a representative cost estimate:
15-day migration = $390.35 + $14.24 × HistoricalTB + $71.86 × LiveMBps
This results in the following cost estimates table:
Note
All costs are rounded to the nearest dollar.
|
15-day full migration cost ($ USD/migration) |
|||
|---|---|---|---|
| Live traffic throughput | |||
| Historical data volume | 5 MBps | 20 MBps | 50 MBps |
| 1 TB | $764 | $1,842 | $4,125 |
| 10 TB | $892 | $1,970 | $3,252 |
| 100 TB | $2,174 | $3,252 | $5,407 |
