.dkr.ecr.us-east-1.amazonaws.com/test_ecr_repository@sha256:52db9000073d93b9bdee6a7246a68c35a741aaade05a8f4febba0bf795cdac02",
"public.ecr.aws/amazonlinux/amazonlinux@sha256:f972d24199508c52de7ad37a298bda35d8a1bd7df158149b381c03f6c6e363b5"
],
"Parent": "",
"Comment": "",
"Created": "2023-02-23T06:20:11.575053226Z",
"Container": "ec7f2fc7d2b6a382384061247ef603e7d647d65f5cd4fa397a3ccbba9278367c",
"ContainerConfig": {
"Hostname": "ec7f2fc7d2b6",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/bash\"]"
],
"Image": "sha256:c1bced1b5a65681e1e0e52d0a6ad17aaf76606149492ca0bf519a466ecb21e51",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"DockerVersion": "20.10.17",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/bash"
],
"Image": "sha256:c1bced1b5a65681e1e0e52d0a6ad17aaf76606149492ca0bf519a466ecb21e51",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 167436755,
"VirtualSize": 167436755,
"GraphDriver": {
"Data": {
"MergedDir": "/var/lib/docker/overlay2/c2c2351a82b26cbdf7782507500e5adb5c2b3a2875bdbba79788a4b27cd6a913/merged",
"UpperDir": "/var/lib/docker/overlay2/c2c2351a82b26cbdf7782507500e5adb5c2b3a2875bdbba79788a4b27cd6a913/diff",
"WorkDir": "/var/lib/docker/overlay2/c2c2351a82b26cbdf7782507500e5adb5c2b3a2875bdbba79788a4b27cd6a913/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:d5655967c2c4e8d68f8ec7cf753218938669e6c16ac1324303c073c736a2e2a2"
]
},
"Metadata": {
"LastTagTime": "2023-03-02T10:28:47.142155987Z"
}
}
]
```
# Install SSM Agent on Amazon EKS worker nodes by using Kubernetes DaemonSet
*Mahendra Revanasiddappa, Amazon Web Services*
## Summary
**Note, September 2021:** The latest Amazon EKS optimized AMIs install SSM Agent automatically. For more information, see the [release notes](https://github.com/awslabs/amazon-eks-ami/releases/tag/v20210621) for the June 2021 AMIs.
In Amazon Elastic Kubernetes Service (Amazon EKS), because of security guidelines, worker nodes don't have Secure Shell (SSH) key pairs attached to them. This pattern shows how you can use the Kubernetes DaemonSet resource type to install AWS Systems Manager Agent (SSM Agent) on all worker nodes, instead of installing it manually or replacing the Amazon Machine Image (AMI) for the nodes. DaemonSet uses a cron job on the worker node to schedule the installation of SSM Agent. You can also use this pattern to install other packages on worker nodes.
When you're troubleshooting issues in the cluster, installing SSM Agent on demand enables you to establish an SSH session with the worker node, to collect logs or to look into instance configuration, without SSH key pairs.
## Prerequisites and limitations
**Prerequisites**
+ An existing Amazon EKS cluster with Amazon Elastic Compute Cloud (Amazon EC2) worker nodes.
+ Container instances should have the required permissions to communicate with the SSM service. The AWS Identity and Access Management (IAM) managed role **AmazonSSMManagedInstanceCore** provides the required permissions for SSM Agent to run on EC2 instances. For more information, see the [AWS Systems Manager documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html).
**Limitations**
+ This pattern isn't applicable to AWS Fargate, because DaemonSets aren't supported on the Fargate platform.
+ This pattern applies only to Linux-based worker nodes.
+ The DaemonSet pods run in privileged mode. If the Amazon EKS cluster has a webhook that blocks pods in privileged mode, the SSM Agent will not be installed.
## Architecture
The following diagram illustrates the architecture for this pattern.
![\[Using Kubernetes DaemonSet to install SSM Agent on Amazon EKS worker nodes.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/016d53f3-45c1-4913-b542-67124e1462b8/images/3a6dfd00-e54b-44d5-843a-4c26ce9826c9.png)
## Tools
**Tools**
+ [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) is a command-line utility that is used to interact with an Amazon EKS cluster. This pattern uses `kubectl` to deploy a DaemonSet on the Amazon EKS cluster, which will install SSM Agent on all worker nodes.
+ [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) makes it easy for you to run Kubernetes on AWS without having to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.
+ [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) lets you manage your EC2 instances, on-premises instances, and virtual machines (VMs) through an interactive, one-click, browser-based shell or through the AWS Command Line Interface (AWS CLI).
**Code**
Use the following code to create a DaemonSet configuration file that will install SSM Agent on the Amazon EKS cluster. Follow the instructions in the [Epics](#install-ssm-agent-on-amazon-eks-worker-nodes-by-using-kubernetes-daemonset-epics) section.
```
cat << EOF > ssm_daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: ssm-installer
name: ssm-installer
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: ssm-installer
template:
metadata:
labels:
k8s-app: ssm-installer
spec:
containers:
- name: sleeper
image: busybox
command: ['sh', '-c', 'echo I keep things running! && sleep 3600']
initContainers:
- image: amazonlinux
imagePullPolicy: Always
name: ssm
command: ["/bin/bash"]
args: ["-c","echo '* * * * * root yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm & rm -rf /etc/cron.d/ssmstart' > /etc/cron.d/ssmstart"]
securityContext:
allowPrivilegeEscalation: true
volumeMounts:
- mountPath: /etc/cron.d
name: cronfile
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumes:
- name: cronfile
hostPath:
path: /etc/cron.d
type: Directory
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
terminationGracePeriodSeconds: 30
EOF
```
## Epics
### Set up kubectl
| Task | Description | Skills required |
| --- | --- | --- |
| Install and configure kubectl to access the EKS cluster. | If `kubectl` isn't already installed and configured to access the Amazon EKS cluster, see [Installing kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) in the Amazon EKS documentation. | DevOps |
### Deploy the DaemonSet
| Task | Description | Skills required |
| --- | --- | --- |
| Create the DaemonSet configuration file. | Use the code in the [Code](#install-ssm-agent-on-amazon-eks-worker-nodes-by-using-kubernetes-daemonset-tools) section earlier in this pattern to create a DaemonSet configuration file called `ssm_daemonset.yaml`, which will be deployed to the Amazon EKS cluster. The pod launched by DaemonSet has a main container and an `init` container. The main container has a `sleep` command. The `init` container includes a `command` section that creates a cron job file to install SSM Agent at the path `/etc/cron.d/`. The cron job runs only once, and the file it creates is automatically deleted after the job is complete. When the init container has finished, the main container waits for 60 minutes before exiting. After 60 minutes, a new pod is launched. This pod installs SSM Agent, if it’s missing, or updates SSM Agent to the latest version.If required, you can modify the `sleep` command to restart the pod once a day or to run more often. | DevOps |
| Deploy the DaemonSet on the Amazon EKS cluster. | To deploy the DaemonSet configuration file you created in the previous step on the Amazon EKS cluster, use the following command:kubectl apply -f ssm_daemonset.yaml
This command creates a DaemonSet to run the pods on worker nodes to install SSM Agent. | DevOps |
## Related resources
+ [Installing kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) (Amazon EKS documentation)
+ [Setting up Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started.html) (AWS Systems Manager documentation)
# Install the SSM Agent and CloudWatch agent on Amazon EKS worker nodes using preBootstrapCommands
*Akkamahadevi Hiremath, Amazon Web Services*
## Summary
This pattern provides code samples and steps to install the AWS Systems Manager Agent (SSM Agent) and Amazon CloudWatch agent on Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes in the Amazon Web Services (AWS) Cloud during Amazon EKS cluster creation. You can install the SSM Agent and CloudWatch agent by using the `preBootstrapCommands` property from the `eksctl` [config file schema](https://eksctl.io/usage/schema/) (Weaveworks documentation). Then, you can use the SSM Agent to connect to your worker nodes without using an Amazon Elastic Compute Cloud (Amazon EC2) key pair. Additionally, you can use the CloudWatch agent to monitor memory and disk utilization on your Amazon EKS worker nodes.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ The [eksctl command line utility](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html), installed and configured on macOS, Linux, or Windows
+ The [kubectl command line utility](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html), installed and configured on macOS, Linux, or Windows
**Limitations**
+ We recommend that you avoid adding long-running scripts to the `preBootstrapCommands`** **property, because this delays the node from joining the Amazon EKS cluster during scaling activities. We recommend that you create a [custom Amazon Machine Image (AMI)](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.customenv.html) instead.
+ This pattern applies to Amazon EC2 Linux instances only.
## Architecture
**Technology stack**
+ Amazon CloudWatch
+ Amazon Elastic Kubernetes Service (Amazon EKS)
+ AWS Systems Manager Parameter Store
**Target architecture**
The following diagram shows an example of a user connecting to Amazon EKS worker nodes using SSM Agent which was installed using the `preBootstrapCommands`.
![\[User connecting to Amazon EKS worker nodes via Systems Manager, with SSM Agent and CloudWatch agent on each node.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/b37a3cdb-204f-4014-8317-3600a793dac7/images/9a5760af-23bb-4616-97b0-b401a9d080cf.png)
The diagram shows the following workflow:
1. The user creates an Amazon EKS cluster by using the `eksctl` configuration file with the `preBootstrapCommands` property, which installs the SSM Agent and CloudWatch agent.
1. Any new instances that join the cluster later due to scaling activities get created with the pre-installed SSM Agent and CloudWatch agent.
1. The user connects to Amazon EC2 by using the SSM Agent and then monitors memory and disk utilization by using the CloudWatch agent.
## Tools
+ [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) helps you monitor the metrics of your AWS resources and the applications that you run on AWS in real time.
+ [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.
+ [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) provides secure, hierarchical storage for configuration data management and secrets management.
+ [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) helps you manage your EC2 instances, on-premises instances, and virtual machines through an interactive, one-click, browser-based shell or through the AWS Command Line Interface (AWS CLI).
+ [eksctl](https://eksctl.io/usage/schema/) is a command-line utility for creating and managing Kubernetes clusters on Amazon EKS.
+ [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) is a command-line utility for communicating with the cluster API server.
## Epics
### Create an Amazon EKS cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Store the CloudWatch agent configuration file. | Store the CloudWatch agent configuration file in the [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) in the AWS Region where you want to create your Amazon EKS cluster. To do this, [create a parameter](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-create-console.html) in AWS Systems Manager Parameter Store and note the name of the parameter (for example, `AmazonCloudwatch-linux`).For more information, see the *Example CloudWatch agent configuration file *code in the [Additional information](#install-the-ssm-agent-and-cloudwatch-agent-on-amazon-eks-worker-nodes-using-prebootstrapcommands-additional) section of this pattern. | DevOps engineer |
| Create the eksctl configuration file and cluster. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/install-the-ssm-agent-and-cloudwatch-agent-on-amazon-eks-worker-nodes-using-prebootstrapcommands.html) | AWS DevOps |
### Verify that the SSM Agent and CloudWatch agent work
| Task | Description | Skills required |
| --- | --- | --- |
| Test the SSM Agent. | Use SSH to connect to your Amazon EKS cluster nodes by using any of the methods covered in [Start a session](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html#start-ec2-console%20%20or%20https:%2F%2Fdocs.aws.amazon.com%2Fsystems-manager%2Flatest%2Fuserguide%2Fsession-manager-working-with-sessions-start.html%23sessions-start-cli) from the AWS Systems Manager documentation. | AWS DevOps |
| Test the CloudWatch agent. | Use the CloudWatch console to validate the CloudWatch agent:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/install-the-ssm-agent-and-cloudwatch-agent-on-amazon-eks-worker-nodes-using-prebootstrapcommands.html) | AWS DevOps |
## Related resources
+ [Installing and running the CloudWatch agent on your servers](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html) (Amazon CloudWatch documentation)
+ [Create a Systems Manager parameter (console)](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-create-console.html) (AWS Systems Manager documentation)
+ [Create the CloudWatch agent configuration file](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file.html) (Amazon CloudWatch documentation)
+ [Starting a session (AWS CLI)](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html#sessions-start-cli) (AWS Systems Manager documentation)
+ [Starting a session (Amazon EC2 console)](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html#start-ec2-console) (AWS Systems Manager documentation)
## Additional information
**Example CloudWatch agent configuration file**
In the following example, the CloudWatch agent is configured to monitor disk and memory utilization on Amazon Linux instances:
```
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "cwagent"
},
"metrics": {
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 60
}
}
}
}
```
**Example eksctl configuration file**
```
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: test
region: us-east-2
version: "1.24"
managedNodeGroups:
- name: test
minSize: 2
maxSize: 4
desiredCapacity: 2
volumeSize: 20
instanceType: t3.medium
preBootstrapCommands:
- sudo yum install amazon-ssm-agent -y
- sudo systemctl enable amazon-ssm-agent
- sudo systemctl start amazon-ssm-agent
- sudo yum install amazon-cloudwatch-agent -y
- sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:AmazonCloudwatch-linux
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
```
**Additional code details**
+ In the last line of the `preBootstrapCommands` property, `AmazonCloudwatch-linux` is the name of the parameter created in AWS System Manager Parameter Store. You must include `AmazonCloudwatch-linux` in Parameter Store in the same AWS Region where you created the Amazon EKS cluster. You can also specify a file path, but we recommend using Systems Manager for easier automation and reusability.
+ If you use `preBootstrapCommands` in the `eksctl` configuration file, you see two launch templates in the AWS Management Console. The first launch template includes the commands specified in `preBootstrapCommands`. The second template includes the commands specified in `preBootstrapCommands` and default Amazon EKS user data. This data is required to get the nodes to join the cluster. The node group’s Auto Scaling group uses this user data to spin up new instances.
+ If you use the `iam` attribute in the `eksctl` configuration file, you must list the default Amazon EKS policies with any additional policies required in your attached AWS Identity and Access Management (IAM) policies. In the code snippet from the *Create the eksctl configuration file and cluster *step, `CloudWatchAgentServerPolicy` and `AmazonSSMMangedInstanceCore` are additional policies added to make sure that the CloudWatch agent and SSM Agent work as expected. The `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, `AmazonEC2ContainerRegistryReadOnly` policies are mandatory policies required for the Amazon EKS cluster to function correctly.
# Migrate NGINX Ingress Controllers when enabling Amazon EKS Auto Mode
*Olawale Olaleye and Shamanth Devagari, Amazon Web Services*
## Summary
[EKS Auto Mode](https://docs.aws.amazon.com/eks/latest/userguide/automode.html) for Amazon Elastic Kubernetes Service (Amazon EKS) can reduce the operational overhead of running your workloads on Kubernetes clusters. This mode allows AWS to also set up and manage the infrastructure on your behalf. When you enable EKS Auto Mode on an existing cluster, you must carefully plan the migration of [NGINX Ingress Controller](https://docs.nginx.com/nginx-ingress-controller/overview/about/) configurations. This is because the direct transfer of Network Load Balancers isn't possible.
You can use a blue/green deployment strategy to migrate an NGINX Ingress Controller instance when you enable EKS Auto Mode in an existing Amazon EKS cluster.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An [Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) running Kubernetes version 1.29 or later
+ Amazon EKS add-ons running [minimum versions](https://docs.aws.amazon.com/eks/latest/userguide/auto-enable-existing.html#auto-addons-required)
+ Latest version of [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html#kubectl-install-update)
+ An existing [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/deploy/#aws) instance
+ (Optional) A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) in Amazon Route 53 for DNS-based traffic shifting
## Architecture
A *blue/green deployment* is a deployment strategy in which you create two separate but identical environments. Blue/green deployments provide near-zero downtime release and rollback capabilities. The fundamental idea is to shift traffic between two identical environments that are running different versions of your application.
The following image shows the migration of Network Load Balancers from two different NGINX Ingress Controller instances when enabling EKS Auto Mode. You use a blue/green deployment to shift traffic between the two Network Load Balancers.
![\[Using a blue/green deployment strategy to migrate NGINX Ingress Controller instances.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/57e8c14f-cb50-4027-8ef6-ce8ea3f2db25/images/211a029a-90d8-4c92-8200-19e54062f936.png)
The original namespace is the *blue* namespace. This is where the original NGINX Ingress Controller service and instance run, before you enable EKS Auto Mode. The original service and instance connect to a Network Load Balancer that has a DNS name that is configured in Route 53. The [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.11/) deployed this Network Load Balancer in the target virtual private cloud (VPC).
The diagram shows the following workflow to set up an environment for a blue/green deployment:
1. Install and configure another NGINX Ingress Controller instance in a different namespace, a *green* namespace.
1. In Route 53, configure a DNS name for a new Network Load Balancer.
## Tools
**AWS services**
+ [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.
+ [Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon Elastic Compute Cloud (Amazon EC2) instances, containers, and IP addresses in one or more Availability Zones.
+ [Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/Welcome.html) is a highly available and scalable DNS web service.
+ [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
**Other tools**
+ [Helm](https://helm.sh/) is an open source package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
+ [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) is a command-line interface that helps you run commands against Kubernetes clusters.
+ [NGINX Ingress Controller](https://docs.nginx.com/nginx-ingress-controller/overview/about/) connects Kubernetes apps and services with request handling, auth, self-service custom resources, and debugging.
## Epics
### Review the existing environment
| Task | Description | Skills required |
| --- | --- | --- |
| Confirm that the original NGINX Ingress Controller instance is operational. | Enter the following command to verify that the resources in the `ingress-nginx` namespace are operational. If you have deployed NGINX Ingress Controller in another namespace, update the namespace name in this command.kubectl get all -n ingress-nginx
In the output, confirm that NGINX Ingress Controller pods are in running state. The following is an example output:NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-xqn9d 0/1 Completed 0 88m
pod/ingress-nginx-admission-patch-lhk4j 0/1 Completed 1 88m
pod/ingress-nginx-controller-68f68f859-xrz74 1/1 Running 2 (10m ago) 72m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.100.67.255 k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com 80:30330/TCP,443:31462/TCP 88m
service/ingress-nginx-controller-admission ClusterIP 10.100.201.176 443/TCP 88m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 88m
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-68f68f859 1 1 1 72m
replicaset.apps/ingress-nginx-controller-d8c96cf68 0 0 0 88m
NAME STATUS COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create Complete 1/1 4s 88m
job.batch/ingress-nginx-admission-patch Complete 1/1 5s 88m
| DevOps engineer |
### Deploy a sample HTTPd workload to use the NGINX Ingress Controller
| Task | Description | Skills required |
| --- | --- | --- |
| Create the Kubernetes resources. | Enter the following commands to create a sample Kubernetes deployment, service, and ingress:kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo
kubectl create ingress demo --class=nginx \
--rule nginxautomode.local.dev/=demo:80
| DevOps engineer |
| Review the deployed resources. | Enter the following command to view a list of the deployed resources:kubectl get all,ingress
In the output, confirm that the sample HTTPd pod is in a running state. The following is an example output:NAME READY STATUS RESTARTS AGE
pod/demo-7d94f8cb4f-q68wc 1/1 Running 0 59m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/demo ClusterIP 10.100.78.155 80/TCP 59m
service/kubernetes ClusterIP 10.100.0.1 443/TCP 117m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/demo 1/1 1 1 59m
NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-7d94f8cb4f 1 1 1 59m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/demo nginx nginxautomode.local.dev k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com 80 56m
| DevOps engineer |
| Confirm the service is reachable. | Enter the following command to confirm that the service is reachable through the DNS name of the Network Load Balancer:curl -H "Host: nginxautomode.local.dev" http://k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com
The following is the expected output:It works!
| DevOps engineer |
| (Optional) Create a DNS record. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-nginx-ingress-controller-eks-auto-mode.html) | DevOps engineer, AWS DevOps |
### Enable EKS Auto Mode on the existing cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Enable EKS Auto Mode. | Follow the instructions in [Enable EKS Auto Mode on an existing cluster](https://docs.aws.amazon.com/eks/latest/userguide/auto-enable-existing.html) (Amazon EKS documentation). | AWS DevOps |
### Install a new NGINX Ingress Controller
| Task | Description | Skills required |
| --- | --- | --- |
| Configure a new NGINX Ingress Controller instance. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-nginx-ingress-controller-eks-auto-mode.html) | DevOps engineer |
| Deploy the new NGINX Instance Controller instance. | Enter the following command to apply the modified manifest file:kubectl apply -f deploy.yaml
| DevOps engineer |
| Confirm successful deployment. | Enter the following command to verify that the resources in the `ingress-nginx-v2` namespace are operational:kubectl get all -n ingress-nginx-v2
In the output, confirm that NGINX Ingress Controller pods are in a running state. The following is an example output:NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-7shrj 0/1 Completed 0 24s
pod/ingress-nginx-admission-patch-vkxr5 0/1 Completed 1 24s
pod/ingress-nginx-controller-757bfcbc6d-4fw52 1/1 Running 0 24s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.100.208.114 k8s-ingressn-ingressn-2e5e37fab6-848337cd9c9d520f.elb.eu-west-1.amazonaws.com 80:31469/TCP,443:30658/TCP 24s
service/ingress-nginx-controller-admission ClusterIP 10.100.150.114 443/TCP 24s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 24s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-757bfcbc6d 1 1 1 24s
NAME STATUS COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create Complete 1/1 4s 24s
job.batch/ingress-nginx-admission-patch Complete 1/1 5s 24s
| DevOps engineer |
| Create a new ingress for the sample HTTPd workload. | Enter the following command to create a new ingress for the existing sample HTTPd workload:kubectl create ingress demo-new --class=nginx-v2 \
--rule nginxautomode.local.dev/=demo:80
| DevOps engineer |
| Confirm that the new ingress works. | Enter the following command to confirm that the new ingress works:curl -H "Host: nginxautomode.local.dev" k8s-ingressn-ingressn-2e5e37fab6-848337cd9c9d520f.elb.eu-west-1.amazonaws.com
The following is the expected output:It works!
| DevOps engineer |
### Cut over
| Task | Description | Skills required |
| --- | --- | --- |
| Cut over to the new namespace. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-nginx-ingress-controller-eks-auto-mode.html) | AWS DevOps, DevOps engineer |
| Review the two ingresses. | Enter the following command to review the two ingresses that were created for the sample HTTPd workload:kubectl get ingress
The following is an example output:NAME CLASS HOSTS ADDRESS PORTS AGE
demo nginx nginxautomode.local.dev k8s-ingressn-ingressn-abcdefg-12345.elb.eu-west-1.amazonaws.com 80 95m
demo-new nginx-v2 nginxautomode.local.dev k8s-ingressn-ingressn-2e5e37fab6-848337cd9c9d520f.elb.eu-west-1.amazonaws.com 80 33s
| DevOps engineer |
## Related resources
+ [Enable EKS Auto Mode on an existing cluster](https://docs.aws.amazon.com/eks/latest/userguide/auto-enable-existing.html) (Amazon EKS documentation)
+ [Troubleshoot load balancers created by the Kubernetes service controller in Amazon EKS](https://repost.aws/knowledge-center/eks-load-balancers-troubleshooting) (AWS re:Post Knowledge Center)
+ [NGINX Ingress Controller](https://docs.nginx.com/nginx-ingress-controller/) (NGINX documentation)
# Migrate your container workloads from Azure Red Hat OpenShift (ARO) to Red Hat OpenShift Service on AWS (ROSA)
*Naveen Ramasamy, Srikanth Rangavajhala, and Gireesh Sreekantan, Amazon Web Services*
## Summary
This pattern provides step-by-step instructions for migrating container workloads from Azure Red Hat OpenShift (ARO) to [Red Hat OpenShift Service on AWS (ROSA)](https://aws.amazon.com/rosa/). ROSA is a managed Kubernetes service provided by Red Hat in collaboration with AWS. It helps you deploy, manage, and scale your containerized applications by using the Kubernetes platform, and benefits from both Red Hat's expertise in Kubernetes and the AWS Cloud infrastructure.
Migrating container workloads from ARO, from other clouds, or from on premises to ROSA involves transferring applications, configurations, and data from one platform to another. This pattern helps ensure a smooth transition while optimizing for AWS Cloud services, security, and cost efficiency. It covers two methods for migrating your workloads to ROSA clusters: CI/CD and Migration Toolkit for Containers (MTC).
This pattern covers both methods. The method you choose depends on the complexity and certainty of your migration process. If you have full control over your application's state and can guarantee a consistent setup through a pipeline, we recommend that you use the CI/CD method. However, if your application state involves uncertainties, unforeseen changes, or a complex ecosystem, we recommend that you use MTC as a reliable and controlled path to migrate your application and its data to a new cluster. For a detailed comparison of the two methods, see the [Additional information](#migrate-container-workloads-from-aro-to-rosa-additional) section.
Benefits of migrating to ROSA:
+ ROSA seamlessly integrates with AWS as a native service. It is easily accessible through the AWS Management Console and billed through a single AWS account. It offers full compatibility with other AWS services and provides collaborative support from both AWS and Red Hat.
+ ROSA supports hybrid and multi-cloud deployments. It enables applications to run consistently across on-premises data centers and multiple cloud environments.
+ ROSA benefits from Red Hat's security focus, and provides features such as role-based access control (RBAC), image scanning, and vulnerability assessments to ensure a secure container environment.
+ ROSA is designed to scale applications easily and provides high availability options. It allows applications to grow as needed while maintaining reliability.
+ ROSA automates and simplifies the deployment of a Kubernetes cluster compared with manual setup and management methods. This accelerates the development and deployment process.
+ ROSA benefits from AWS Cloud services, and provides seamless integration with AWS offerings such as database services, storage solutions, and security services.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ Permissions configured for AWS services that ROSA relies on to deliver functionality. For more information, see [Prerequisites](https://docs.aws.amazon.com/rosa/latest/userguide/set-up.html) in the ROSA documentation.
+ ROSA enabled on the [ROSA console](https://console.aws.amazon.com/rosa). For instructions, see the [ROSA documentation](https://docs.aws.amazon.com/rosa/latest/userguide/set-up.html#enable-rosa).
+ The ROSA cluster installed and configured. For more information, see [Get started with ROSA](https://docs.aws.amazon.com/rosa/latest/userguide/getting-started.html) in the ROSA documentation. To understand the different methods for setting up a ROSA cluster, see the AWS Prescriptive Guidance guide [ROSA implementation strategies](https://docs.aws.amazon.com/prescriptive-guidance/latest/red-hat-openshift-on-aws-implementation/).
+ Network connectivity established from the on-premises network to AWS through [AWS Direct Connect](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/aws-direct-connect.html) (preferred) or [AWS Virtual Private Network (Site-to-Site VPN)](https://docs.aws.amazon.com/vpc/latest/userguide/vpn-connections.html).
+ An Amazon Elastic Compute Cloud (Amazon EC2) instance or another virtual server to install tools such as `aws client`, OpenShift CLI (`oc`) client, ROSA client, and Git binary.
Additional prerequisites for the CI/CD method:
+ Access to the on-premises Jenkins server with permissions to create a new pipeline, add stages, add OpenShift clusters, and perform builds.
+ Access to the Git repository where application source code is maintained, with permissions to create a new Git branch and perform commits to the new branch.
Additional prerequisites for the MTC method:
+ An Amazon Simple Storage Service (Amazon S3) bucket, which will be used as a replication repository.
+ Administrative access to the source ARO cluster. This is required to set up the MTC connection.
**Limitations**
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
## Architecture
ROSA provides three network deployment patterns: public, private, and [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html). PrivateLink enables Red Hat site reliability engineering (SRE) teams to manage the cluster by using a private subnet connected to the cluster’s PrivateLink endpoint in an existing VPC.
Choosing the PrivateLink option provides the most secure configuration. For that reason, we recommend it for sensitive workloads or strict compliance requirements. For information about the public and private network deployment options, see the [Red Hat OpenShift documentation](https://docs.openshift.com/rosa/architecture/rosa-architecture-models.html#rosa-hcp-architecture_rosa-architecture-models).
**Important**
You can create a PrivateLink cluster only at installation time. You cannot change a cluster to use PrivateLink after installation.
The following diagram illustrates the PrivateLink architecture for a ROSA cluster that uses Direct Connect to connect to the on-premises and ARO environments.
![\[ROSA cluster that uses AWS Direct Connect and AWS PrivateLink.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/527cedfb-ec21-42be-bf21-d4e4e4f9db51/images/eff9b017-6fc7-4874-b610-849a42071ef4.png)
**AWS permissions to ROSA**
For AWS permissions to ROSA, we recommend that you use AWS Security Token Service (AWS STS) with short-lived, dynamic tokens. This method uses least-privilege predefined roles and policies to grant ROSA minimal permissions to operate in the AWS account, and supports ROSA installation, control plane, and compute functionality.
**CI/CD pipeline redeployment**
CI/CD pipeline redeployment is the recommended method for users who have a mature CI/CD pipeline. When you choose this option, you can use any [DevOps deployment strategy ](https://docs.aws.amazon.com/whitepapers/latest/introduction-devops-aws/deployment-strategies.html)to gradually shift your application load to deployments on ROSA.
**Note**
This pattern assumes a common use case where you have an on-premises Git, JFrog Artifactory, and Jenkins pipeline. This approach requires that you establish network connectivity from your on-premises network to AWS through Direct Connect, and that you set up the ROSA cluster before you follow the instructions in the [Epics](#migrate-container-workloads-from-aro-to-rosa-epics) section. See the [Prerequisites](#migrate-container-workloads-from-aro-to-rosa-prereqs) section for details.
The following diagram shows the workflow for this method.
![\[Migrating containers from ARO to ROSA by using the CI/CD method.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/527cedfb-ec21-42be-bf21-d4e4e4f9db51/images/f658590e-fbd9-4297-a02c-0b516694d436.png)
**MTC method**
You can use the [Migration Toolkit for Containers (MTC)](https://docs.openshift.com/container-platform/4.13/migration_toolkit_for_containers/about-mtc.html)** **to migrate your containerized workloads between different Kubernetes environments, such as from ARO to ROSA. MTC simplifies the migration process by automating several key tasks and providing a comprehensive framework for managing the migration lifecycle.
The following diagram shows the workflow for this method.
![\[Migrating containers from ARO to ROSA by using the MTC method.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/527cedfb-ec21-42be-bf21-d4e4e4f9db51/images/979bbc7b-2e39-4dd1-b4f0-ea1032880a38.png)
## Tools
**AWS services**
+ [AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html) is an online data transfer and discovery service that helps you move files or object data to, from, and between AWS storage services.
+ [AWS Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable. With this connection, you can create virtual interfaces directly to public AWS services while bypassing internet service providers in your network path.
+ [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) helps you create unidirectional, private connections from your virtual private clouds (VPCs) to services outside of the VPC.
+ [Red Hat OpenShift Service on AWS (ROSA)](https://docs.aws.amazon.com/rosa/latest/userguide/what-is-rosa.html) is a managed service that helps Red Hat OpenShift users to build, scale, and manage containerized applications on AWS.
+ [AWS Security Token Service (AWS STS)](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) helps you request temporary, limited-privilege credentials for users.
**Other tools**
+ [Migration Toolkit for Containers (MTC)](https://docs.openshift.com/container-platform/4.13/migration_toolkit_for_containers/about-mtc.html) provides a console and API for migrating containerized applications from ARO to ROSA.
## Best practices
+ For [resilience](https://docs.aws.amazon.com/ROSA/latest/userguide/disaster-recovery-resiliency.html) and if you have security compliance workloads, set up a Multi-AZ ROSA cluster that uses PrivateLink. For more information, see the [ROSA documentation](https://docs.aws.amazon.com/rosa/latest/userguide/getting-started-classic-private-link.html).
**Note**
PrivateLink cannot be configured after installation.
+ The S3 bucket that you use for replication repository should not be public. Use the appropriate S3 bucket policies to restrict access.
+ If you choose the MTC method, use the **Stage** migration option to reduce the downtime window during cutover.
+ Review your service quotas before and after you provision the ROSA cluster. If necessary, request a quota increase according to your requirements. For more information, see the [Service Quotas documentation](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html).
+ Review the [ROSA security guidelines](https://docs.aws.amazon.com/ROSA/latest/userguide/security.html) and implement security best practices.
+ We recommend that you remove the default cluster administrator after installation. For more information, see the [Red Hat OpenShift documentation](https://docs.openshift.com/container-platform/4.13/post_installation_configuration/cluster-tasks.html).
+ Use machine pool automatic scaling to scale down unused worker nodes in the ROSA cluster to optimize costs. For more information, see the [ROSA Workshop](https://catalog.workshops.aws/aws-openshift-workshop/en-US/5-nodes-storage/3-autoscale-machine-pool).
+ Use the Red Hat Cost Management service for OpenShift Container Platform to better understand and track costs for clouds and containers. For more information, see the [ROSA Workshop](https://catalog.workshops.aws/aws-openshift-workshop/en-US/10-cost-management).
+ Monitor and audit ROSA cluster infrastructure services and applications by using AWS services. For more information, see the [ROSA Workshop](https://catalog.workshops.aws/aws-openshift-workshop/en-US/8-observability).
## Epics
### Option 1: Use a CI/CD pipeline
| Task | Description | Skills required |
| --- | --- | --- |
| Add the new ROSA cluster to Jenkins. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-container-workloads-from-aro-to-rosa.html) | AWS administrator, AWS systems administrator, AWS DevOps |
| Add the `oc` client to your Jenkins nodes. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-container-workloads-from-aro-to-rosa.html) | AWS administrator, AWS systems administrator, AWS DevOps |
| Create a new Git branch. | Create a new branch in your Git repository for `rosa-dev`. This branch separates the code or configuration parameter changes for ROSA from your existing pipeline. | AWS DevOps |
| Tag images for ROSA. | In your build stage, use a different tag to identify the images that are built from the ROSA pipeline. | AWS administrator, AWS systems administrator, AWS DevOps |
| Create a pipeline. | Create a new Jenkins pipeline that's similar to your existing pipeline. For this pipeline, use the `rosa-dev` Git branch that you created earlier, and make sure to include the Git checkout, code scan, and build stages that are identical to your existing pipeline. | AWS administrator, AWS systems administrator, AWS DevOps |
| Add a ROSA deployment stage. | In the new pipeline, add a stage to deploy to the ROSA cluster and reference the ROSA cluster that you added to the Jenkins global configuration. | AWS administrator, AWS DevOps, AWS systems administrator |
| Start a new build. | In Jenkins, select your pipeline and choose **Build now**, or start a new build by committing a change to the `rosa-dev` branch in Git. | AWS administrator, AWS DevOps, AWS systems administrator |
| Verify the deployment. | Use the **oc** command or the [ROSA console](https://console.aws.amazon.com/rosa) to verify that the application has been deployed on your target ROSA cluster. | AWS administrator, AWS DevOps, AWS systems administrator |
| Copy data to the target cluster. | For stateful workloads, copy the data from the source cluster to the target cluster by using AWS DataSync or open source utilities such as **rsync**, or you can use the MTC method. For more information, see the [AWS DataSync documentation](https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html). | AWS administrator, AWS DevOps, AWS systems administrator |
| Test your application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-container-workloads-from-aro-to-rosa.html) | AWS administrator, AWS DevOps, AWS systems administrator |
| Cut over. | If your testing is successful, use the appropriate Amazon Route 53 policy to move the traffic from the ARO-hosted application to the ROSA-hosted application. When you complete this step, your application's workload will fully transition to the ROSA cluster. | AWS administrator, AWS systems administrator |
### Option 2: Use MTC
| Task | Description | Skills required |
| --- | --- | --- |
| Install the MTC operator. | Install the MTC operator on both ARO and ROSA clusters:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-container-workloads-from-aro-to-rosa.html) | AWS administrator, AWS DevOps, AWS systems administrator |
| Configure network traffic to the replication repository. | If you're using a proxy server, configure it to allow network traffic between the replication repository and the clusters. The replication repository is an intermediate storage object that MTC uses to migrate data. The source and target clusters must have network access to the replication repository during migration. | AWS administrator, AWS DevOps, AWS systems administrator |
| Add the source cluster to MTC. | On the MTC web console, add the ARO source cluster. | AWS administrator, AWS DevOps, AWS systems administrator |
| Add Amazon S3 as your replication repository. | On the MTC web console, add the Amazon S3 bucket (see [Prerequisites](#migrate-container-workloads-from-aro-to-rosa-prereqs)) as the replication repository. | AWS administrator, AWS DevOps, AWS systems administrator |
| Create a migration plan. | On the MTC web console, create a migration plan and specify the data transfer type as **Copy**. This will instruct MTC to copy the data from the source (ARO) cluster to the S3 bucket, and from the bucket to the target (ROSA) cluster. | AWS administrator, AWS DevOps, AWS systems administrator |
| Run the migration plan. | Run the migration plan by using the **Stage** or **Cutover** option:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-container-workloads-from-aro-to-rosa.html) | AWS administrator, AWS DevOps, AWS systems administrator |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| Connectivity issues | When you migrate your container workloads from ARO to ROSA, you might encounter connectivity issues that should be resolved to ensure a successful migration. To address these connectivity issues (listed in this table) during migration, meticulous planning, coordination with your network and security teams, and thorough testing are vital. Implementing a gradual migration strategy and verifying connectivity at each step will help minimize potential disruptions and ensure a smooth transition from ARO to ROSA. |
| Network configuration differences | ARO and ROSA might have variations in their network configurations, such as virtual network (VNet) settings, subnets, and network policies. For proper communication between services, make sure that the network settings align between the two platforms. |
| Security group and firewall rules | ROSA and ARO might have different default security group and firewall settings. Make sure to adjust and update these rules to permit necessary traffic to maintain connectivity among containers and services during migration. |
| IP address and DNS changes | When you migrate workloads, IP addresses and DNS names might change. Reconfigure applications that rely on static IPs or specific DNS names. |
| External service access | If your application depends on external services such as databases or APIs, you might have to update their connection settings to make sure they can communicate with the new services from ROSA. |
| Azure Private Link configuration | If you use Azure Private Link or private endpoint services in ARO, you will need to set up the equivalent functionality in ROSA to ensure private connectivity between resources. |
| Site-to-Site VPN or Direct Connect setup | If there are existing Site-to-Site VPN or Direct Connect connections between your on-premises network and ARO, you will need to establish similar connections with ROSA for uninterrupted communication with your local resources. |
| Ingress and load balancer settings | Configurations for ingress controllers and load balancers might differ between ARO and ROSA. Reconfigure these settings to maintain external access to your services. |
| Certificate and TLS handling | If your applications use SSL certificates or TLS, make sure that the certificates are valid and configured correctly in ROSA. |
| Container registry access | If your containers are hosted in an external container registry, set up the proper authentication and access permissions for ROSA. |
| Monitoring and logging | Update monitoring and logging configurations to reflect the new infrastructure on ROSA so you can continue to monitor the health and performance of your containers effectively. |
## Related resources
**AWS**** references**
+ [What is Red Hat OpenShift Service on AWS?](https://docs.aws.amazon.com/ROSA/latest/userguide/what-is-rosa.html) (ROSA documentation)
+ [Get started with ROSA](https://docs.aws.amazon.com/ROSA/latest/userguide/getting-started.html) (ROSA documentation)
+ [Red Hat OpenShift Service on AWS implementation strategies](https://docs.aws.amazon.com/prescriptive-guidance/latest/red-hat-openshift-on-aws-implementation/) (AWS Prescriptive Guidance)
+ [Red Hat OpenShift Service on AWS Now GA](https://aws.amazon.com/blogs/aws/red-hat-openshift-service-on-aws-now-generally-availably/) (AWS blog post)
+ [ROSA Workshop](https://catalog.workshops.aws/aws-openshift-workshop/en-US/0-introduction)
+ [ROSA FAQ](https://aws.amazon.com/rosa/faqs/)
+ [ROSA Workshop FAQ](https://www.rosaworkshop.io/rosa/14-faq/)
+ [ROSA pricing](https://aws.amazon.com/rosa/pricing/)
**Red Hat OpenShift documentation**
+ [Installing a cluster quickly on AWS](https://docs.openshift.com/container-platform/4.13/installing/installing_aws/installing-aws-default.html)
+ [Installing a cluster on AWS in a restricted network](https://docs.openshift.com/container-platform/4.13/installing/installing_aws/installing-restricted-networks-aws-installer-provisioned.html)
+ [Installing a cluster on AWS into an existing VPC](https://docs.openshift.com/container-platform/4.13/installing/installing_aws/installing-aws-vpc.html)
+ [Installing a cluster on user-provisioned infrastructure in AWS by using CloudFormation templates](https://docs.openshift.com/container-platform/4.13/installing/installing_aws/installing-aws-user-infra.html)
+ [Installing a cluster on AWS in a restricted network with user-provisioned infrastructure](https://docs.openshift.com/container-platform/4.13/installing/installing_aws/installing-restricted-networks-aws.html)
+ [Installing a cluster on AWS with customizations](https://docs.openshift.com/container-platform/4.13/installing/installing_aws/installing-aws-customizations.html)
+ [Getting started with the OpenShift CLI](https://docs.openshift.com/container-platform/4.13/cli_reference/openshift_cli/getting-started-cli.html)
## Additional information
**Choosing between MTC and CI/CD pipeline redeployment options**
Migrating applications from one OpenShift cluster to another requires careful consideration. Ideally, you would want a smooth transition by using a CI/CD pipeline to redeploy the application and handle the migration of persistent volume data. However, in practice, a running application on a cluster is susceptible to unforeseen changes over time. These changes can cause the application to gradually deviate from its original deployment state. MTC offers a solution for scenarios where the exact contents of a namespace are uncertain and a seamless migration of all application components to a new cluster is important.
Making the right choice requires evaluating your specific scenario and weighing the benefits of each approach. By doing so, you can ensure a successful and seamless migration that aligns with your needs and priorities. Here are additional guidelines for choosing between the two options.
**CI/CD pipeline redeployment**
The CI/CD pipeline method is the recommended approach if your application can be confidently redeployed by using a pipeline. This ensures that the migration is controlled, predictable, and aligned with your existing deployment practices. When you choose this method, you can use [blue/green deployment](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/bluegreen-deployments.html) or canary deployment strategies to gradually shift the load to deployments on ROSA. For this scenario, this pattern assumes that Jenkins is orchestrating application deployments from the on-premises environment.
Advantages:
+ You do not require administrative access to the source ARO cluster or need to deploy any operators on the source or destination cluster.
+ This approach helps you switch traffic gradually by using a DevOps strategy.
Disadvantages:
+ It requires more effort to test the functionality of your application.
+ If your application contains persistent data, it requires additional steps to copy the data by using AWS DataSync or other tools.
**MTC migration**
In the real world, running applications can undergo unanticipated changes that cause them to drift away from the initial deployment. Choose the MTC option when you're unsure about the current state of your application on the source cluster. For example, if your application ecosystem spans various components, configurations, and data storage volumes, we recommend that you choose MTC to ensure a complete migration that covers the application and its entire environment.
Advantages:
+ MTC provides complete backup and restore of the workload.
+ It will copy the persistent data from source to target while migrating the workload.
+ It does not require access to the source code repository.
Disadvantages:
+ You need administrative privileges to install the MTC operator on the source and destination clusters.
+ The DevOps team requires training to use the MTC tool and perform migrations.
# Run Amazon ECS tasks on Amazon WorkSpaces with Amazon ECS Anywhere
*Akash Kumar, Amazon Web Services*
## Summary
Amazon Elastic Container Service (Amazon ECS) Anywhere supports the deployment of Amazon ECS tasks in any environment, including Amazon Web Services (AWS) managed infrastructure and customer managed infrastructure. You can do this while using a fully AWS managed control plane that’s running in the cloud and always up to date.
Enterprises often use Amazon WorkSpaces for developing container-based applications. This has required Amazon Elastic Compute Cloud (Amazon EC2) or AWS Fargate with an Amazon ECS cluster to test and run ECS tasks. Now, by using Amazon ECS Anywhere, you can add Amazon WorkSpaces as external instances directly to an ECS cluster, and you can run your tasks directly. This reduces your development time, because you can test your container with an ECS cluster locally on Amazon WorkSpaces. You can also save the cost of using EC2 or Fargate instances for testing your container applications.
This pattern showcases how to deploy ECS tasks on Amazon WorkSpaces with Amazon ECS Anywhere. It sets up the ECS cluster and uses AWS Directory Service Simple AD to launch the WorkSpaces. Then the example ECS task launches NGINX in the WorkSpaces.
## Prerequisites and limitations
+ An active AWS account
+ AWS Command Line Interface (AWS CLI)
+ AWS credentials [configured on your machine](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
## Architecture
**Target technology stack**
+ A virtual private cloud (VPC)
+ An Amazon ECS cluster
+ Amazon WorkSpaces
+ AWS Directory Service with Simple AD
**Target architecture **
![\[ECS Anywhere sets up ECS cluster and uses Simple AD to launch WorkSpaces.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/da8b2249-3423-485c-9fef-6f902025e969/images/fd354d14-f29b-4b9e-8f1a-c3cb7ed4d6bf.png)
The architecture includes the following services and resources:
+ An ECS cluster with public and private subnets in a custom VPC
+ Simple AD in the VPC to provide user access to Amazon WorkSpaces
+ Amazon WorkSpaces provisioned in the VPC using Simple AD
+ AWS Systems Manager activated for adding Amazon WorkSpaces as managed instances
+ Using Amazon ECS and AWS Systems Manager Agent (SSM Agent), Amazon WorkSpaces added to Systems Manager and the ECS cluster
+ An example ECS task to run in the WorkSpaces in the ECS cluster
## Tools
+ [AWS Directory Service Simple Active Directory (Simple AD)](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_simple_ad.html) is a standalone managed directory powered by a Samba 4 Active Directory Compatible Server. Simple AD provides a subset of the features offered by AWS Managed Microsoft AD, including the ability to manage users and to securely connect to Amazon EC2 instances.
+ [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a fast and scalable container management service that helps you run, stop, and manage containers on a cluster.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) helps you manage your applications and infrastructure running in the AWS Cloud. It simplifies application and resource management, shortens the time to detect and resolve operational problems, and helps you manage your AWS resources securely at scale.
+ [Amazon WorkSpaces](https://docs.aws.amazon.com/workspaces/latest/adminguide/amazon-workspaces.html) helps you provision virtual, cloud-based Microsoft Windows or Amazon Linux desktops for your users, known as *WorkSpaces*. WorkSpaces eliminates the need to procure and deploy hardware or install complex software.
## Epics
### Set up the ECS cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Create and configure the ECS cluster. | To create the ECS cluster, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create_cluster.html), including the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-amazon-ecs-tasks-on-amazon-workspaces-with-amazon-ecs-anywhere.html) | Cloud architect |
### Launch Amazon WorkSpaces
| Task | Description | Skills required |
| --- | --- | --- |
| Set up Simple AD and launch Amazon WorkSpaces. | To provision a Simple AD directory for your newly created VPC and launch Amazon WorkSpaces, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/workspaces/latest/adminguide/launch-workspace-simple-ad.html). | Cloud architect |
### Set up AWS Systems Manager for a hybrid environment
| Task | Description | Skills required |
| --- | --- | --- |
| Download the attached scripts. | On your local machine, download the `ssm-trust-policy.json` and `ssm-activation.json` files that are in the *Attachments* section. | Cloud architect |
| Add the IAM role. | Add environment variables based on your business requirements.export AWS_DEFAULT_REGION=${AWS_REGION_ID}
export ROLE_NAME=${ECS_TASK_ROLE}
export CLUSTER_NAME=${ECS_CLUSTER_NAME}
export SERVICE_NAME=${ECS_CLUSTER_SERVICE_NAME}Run the following command.aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://ssm-trust-policy.json
| Cloud architect |
| Add the AmazonSSMManagedInstanceCore policy to the IAM role. | Run the following command.aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
| Cloud architect |
| Add the AmazonEC2ContainerServiceforEC2Role policy to IAM role. | Run the following command.aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
| Cloud architect |
| Verify the IAM role. | To verify the IAM role, run the following command.aws iam list-attached-role-policies --role-name $ROLE_NAME
| Cloud architect |
| Activate Systems Manager. | Run the following command.aws ssm create-activation --iam-role $ROLE_NAME | tee ssm-activation.json
| Cloud architect |
### Add WorkSpaces to the ECS cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Connect to your WorkSpaces. | To connect to and set up your Workspaces, follow the instructions in the [AWS documentation](https://docs.aws.amazon.com/workspaces/latest/userguide/workspaces-user-getting-started.html). | App developer |
| Download the ecs-anywhere install script. | At the command prompt, run the following command.curl -o "ecs-anywhere-install.sh" "https://amazon-ecs-agent-packages-preview.s3.us-east-1.amazonaws.com/ecs-anywhere-install.sh" && sudo chmod +x ecs-anywhere-install.sh
| App developer |
| Check integrity of the shell script. | (Optional) Run the following command.curl -o "ecs-anywhere-install.sh.sha256" "https://amazon-ecs-agent-packages-preview.s3.us-east-1.amazonaws.com/ecs-anywhere-install.sh.sha256" && sha256sum -c ecs-anywhere-install.sh.sha256
| App developer |
| Add an EPEL repository on Amazon Linux. | To add an Extra Packages for Enterprise Linux (EPEL) repository, run the command `sudo amazon-linux-extras install epel -y`. | App developer |
| Install Amazon ECS Anywhere. | To run the install script, use the following command.sudo ./ecs-anywhere-install.sh --cluster $CLUSTER_NAME --activation-id $ACTIVATION_ID --activation-code $ACTIVATION_CODE --region $AWS_REGION
| |
| Check instance information from the ECS cluster. | To check the Systems Manager and ECS cluster instance information and validate that WorkSpaces were added on the cluster, run the following command from your local machine.aws ssm describe-instance-information" && "aws ecs list-container-instances --cluster $CLUSTER_NAME
| App developer |
### Add an ECS task for the WorkSpaces
| Task | Description | Skills required |
| --- | --- | --- |
| Create a task execution IAM role. | Download `task-execution-assume-role.json` and `external-task-definition.json` from the *Attachments* section. On your local machine, run the following command.aws iam --region $AWS_DEFAULT_REGION create-role --role-name $ECS_TASK_EXECUTION_ROLE --assume-role-policy-document file://task-execution-assume-role.json
| Cloud architect |
| Add the policy to the execution role. | Run the following command.aws iam --region $AWS_DEFAULT_REGION attach-role-policy --role-name $ECS_TASK_EXECUTION_ROLE --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
| Cloud architect |
| Create a task role. | Run the following command.aws iam --region $AWS_DEFAULT_REGION create-role --role-name $ECS_TASK_EXECUTION_ROLE --assume-role-policy-document file://task-execution-assume-role.json
| Cloud architect |
| Register the task definition to the cluster. | On your local machine, run the following command.aws ecs register-task-definition --cli-input-json file://external-task-definition.json
| Cloud architect |
| Run the task. | On your local machine, run the following command.aws ecs run-task --cluster $CLUSTER_NAME --launch-type EXTERNAL --task-definition nginx
| Cloud architect |
| Validate the task running state. | To fetch the task ID, run the following command.export TEST_TASKID=$(aws ecs list-tasks --cluster $CLUSTER_NAME | jq -r '.taskArns[0]')
With the task ID, run the following command.aws ecs describe-tasks --cluster $CLUSTER_NAME --tasks ${TEST_TASKID} | Cloud architect |
| Verify the task on the WorkSpace. | To check that NGINX is running on the WorkSpace, run the command` curl http://localhost:8080`. | App developer |
## Related resources
+ [ECS clusters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html)
+ [Setting up a hybrid environment](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-managedinstances.html)
+ [Amazon WorkSpaces](https://docs.aws.amazon.com/workspaces/latest/adminguide/amazon-workspaces.html)
+ [Simple AD](https://docs.aws.amazon.com/workspaces/latest/adminguide/launch-workspace-simple-ad.html)
## Attachments
To access additional content that is associated with this document, unzip the following file: [attachment.zip](samples/p-attach/da8b2249-3423-485c-9fef-6f902025e969/attachments/attachment.zip)
# Run an ASP.NET Core web API Docker container on an Amazon EC2 Linux instance
*Vijai Anand Ramalingam and Sreelaxmi Pai, Amazon Web Services*
## Summary
This pattern is for people who are starting to containerize their applications on the Amazon Web Services (AWS) Cloud. When you begin to containerize apps on cloud, usually there are no container orchestrating platforms set up. This pattern helps you quickly set up infrastructure on AWS to test your containerized applications without needing an elaborate container orchestrating infrastructure.
The first step in the modernization journey is to transform the application. If it's a legacy .NET Framework application, you must first change the runtime to ASP.NET Core. Then do the following:
+ Create the Docker container image
+ Run the Docker container using the built image
+ Validate the application before deploying it on any container orchestration platform, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS).
This pattern covers the build, run, and validate aspects of modern application development on an Amazon Elastic Compute Cloud (Amazon EC2) Linux instance.
## Prerequisites and limitations
**Prerequisites **
+ An active [Amazon Web Services (AWS) account](https://aws.amazon.com/account/)
+ An [AWS Identity and Access Management (IAM) role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) with sufficient access to create AWS resources for this pattern
+ [Visual Studio Community 2022](https://visualstudio.microsoft.com/downloads/) or later downloaded and installed
+ A .NET Framework project modernized to ASP.NET Core
+ A GitHub repository
**Product versions**
+ Visual Studio Community 2022 or later
## Architecture
**Target architecture **
This pattern uses an [AWS CloudFormation template](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=SSM-SSH-Demo&templateURL=https://aws-quickstart.s3.amazonaws.com/quickstart-examples/samples/session-manager-ssh/session-manager-example.yaml) to create the highly available architecture shown in the following diagram. An Amazon EC2 Linux instance is launched in a private subnet. AWS Systems Manager Session Manager is used to access the private Amazon EC2 Linux instance and to test the API running in the Docker container.
![\[A user accessing the Amazon EC2 Linux instance and testing the API running in the Docker container.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/512e61b2-10ba-43be-bbd8-2bdc597c3de3/images/9c5206f6-32b1-47be-9037-360c0bff713c.png)
1. Access to the Linux instance through Session Manager
## Tools
**AWS services**
+ [AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) – AWS Command Line Interface (AWS CLI) is an open source tool for interacting with AWS services through commands in your command line shell. With minimal configuration, you can run AWS CLI commands that implement functionality equivalent to that provided by the browser-based AWS Management Console.
+ [AWS Management Console](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/learn-whats-new.html) – The AWS Management Console is a web application that comprises and refers to a broad collection of service consoles for managing AWS resources. When you first sign in, you see the console home page. The home page provides access to each service console and offers a single place to access the information you need to perform your AWS related tasks.
+ [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) – Session Manager is a fully managed AWS Systems Manager capability. With Session Manager, you can manage your Amazon Elastic Compute Cloud (Amazon EC2) instances. Session Manager provides secure and auditable node management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys.
**Other tools**
+ [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/) – Visual Studio 2022 is an integrated development environment (IDE).
+ [Docker](https://www.docker.com/) – Docker is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers.
**Code**
```
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["DemoNetCoreWebAPI/DemoNetCoreWebAPI.csproj", "DemoNetCoreWebAPI/"]
RUN dotnet restore "DemoNetCoreWebAPI/DemoNetCoreWebAPI.csproj"
COPY . .
WORKDIR "/src/DemoNetCoreWebAPI"
RUN dotnet build "DemoNetCoreWebAPI.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "DemoNetCoreWebAPI.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "DemoNetCoreWebAPI.dll"]
```
## Epics
### Develop the ASP.NET Core web API
| Task | Description | Skills required |
| --- | --- | --- |
| Create an example ASP.NET Core web API using Visual Studio. | To create an example ASP.NET Core web API, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html) | App developer |
| Create a Dockerfile. | To create a Dockerfile, do one of the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html)To push the changes to your GitHub repository, run the following command.git add --all
git commit -m "Dockerfile added"
git push
| App developer |
### Set up the Amazon EC2 Linux instance
| Task | Description | Skills required |
| --- | --- | --- |
| Set up the infrastructure. | Launch the [AWS CloudFormation template](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=SSM-SSH-Demo&templateURL=https://aws-quickstart.s3.amazonaws.com/quickstart-examples/samples/session-manager-ssh/session-manager-example.yaml) to create the infrastructure, which includes the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html)To learn more about accessing a private Amazon EC2 instance using Session Manager without requiring a bastion host, see the [Toward a bastion-less world](https://aws.amazon.com/blogs/infrastructure-and-automation/toward-a-bastion-less-world/) blog post. | App developer, AWS administrator, AWS DevOps |
| Log in to the Amazon EC2 Linux instance. | To connect to the Amazon EC2 Linux instance in the private subnet, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html) | App developer |
| Install and start Docker. | To install and start Docker in the Amazon EC2 Linux instance, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html) | App developer, AWS administrator, AWS DevOps |
| Install Git and clone the repository. | To install Git on the Amazon EC2 Linux instance and clone the repository from GitHub, do the following.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html) | App developer, AWS administrator, AWS DevOps |
| Build and run the Docker container. | To build the Docker image and run the container inside the Amazon EC2 Linux instance, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-an-asp-net-core-web-api-docker-container-on-an-amazon-ec2-linux-instance.html) | App developer, AWS administrator, AWS DevOps |
### Test the web API
| Task | Description | Skills required |
| --- | --- | --- |
| Test the web API using the curl command. | To test the web API, run the following command.curl -X GET "http://localhost/WeatherForecast" -H "accept: text/plain"
Verify the API response.You can get the curl commands for each endpoint from Swagger when you are running it locally. | App developer |
### Clean up resources
| Task | Description | Skills required |
| --- | --- | --- |
| Delete all resources. | Delete the stack to remove all the resources. This ensures that you aren’t charged for any services that you aren’t using. | AWS administrator, AWS DevOps |
## Related resources
+ [Connect to your Linux instance from Windows using PuTTY](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html)
+ [Create a web API with ASP.NET Core](https://docs.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-5.0&tabs=visual-studio)
+ [Toward a bastion-less world](https://aws.amazon.com/blogs/infrastructure-and-automation/toward-a-bastion-less-world/)
# Run stateful workloads with persistent data storage by using Amazon EFS on Amazon EKS with AWS Fargate
*Ricardo Morais, Rodrigo Bersa, and Lucio Pereira, Amazon Web Services*
## Summary
This pattern provides guidance for enabling Amazon Elastic File System (Amazon EFS) as a storage device for containers that are running on Amazon Elastic Kubernetes Service (Amazon EKS) by using AWS Fargate to provision your compute resources.
The setup described in this pattern follows security best practices and provides security at rest and security in transit by default. To encrypt your Amazon EFS file system, it uses an AWS Key Management Service (AWS KMS) key, but you can also specify a key alias that dispatches the process of creating a KMS key.
You can follow the steps in this pattern to create a namespace and Fargate profile for a proof-of-concept (PoC) application, install the Amazon EFS Container Storage Interface (CSI) driver that is used to integrate the Kubernetes cluster with Amazon EFS, configure the storage class, and deploy the PoC application. These steps result in an Amazon EFS file system that is shared among multiple Kubernetes workloads, running over Fargate. The pattern is accompanied by scripts that automate these steps.
You can use this pattern if you want data persistence in your containerized applications and want to avoid data loss during scaling operations. For example:
+ **DevOps tools** – A common scenario is to develop a continuous integration and continuous delivery (CI/CD) strategy. In this case, you can use Amazon EFS as a shared file system to store configurations among different instances of the CI/CD tool or to store a cache (for example, an Apache Maven repository) for pipeline stages among different instances of the CI/CD tool.
+ **Web servers** – A common scenario is to use Apache as an HTTP web server. You can use Amazon EFS as a shared file system to store static files that are shared among different instances of the web server. In this example scenario, modifications are applied directly to the file system instead of static files being baked into a Docker image.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account
+ An existing Amazon EKS cluster with Kubernetes version 1.17 or later (tested up to version 1.27)
+ An existing Amazon EFS file system to bind a Kubernetes StorageClass and provision file systems dynamically
+ Cluster administration permissions
+ Context configured to point to the desired Amazon EKS cluster
**Limitations**
+ There are some limitations to consider when you’re using Amazon EKS with Fargate. For example, the use of some Kubernetes constructs, such as DaemonSets and privileged containers, aren’t supported. For more information, about Fargate limitations, see the [AWS Fargate considerations](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html#fargate-considerations) in the Amazon EKS documentation.
+ The code provided with this pattern supports workstations that are running Linux or macOS.
**Product versions**
+ AWS Command Line Interface (AWS CLI) version 2 or later
+ Amazon EFS CSI driver version 1.0 or later (tested up to version 2.4.8)
+ eksctl version 0.24.0 or later (tested up to version 0.158.0)
+ jq version 1.6 or later
+ kubectl version 1.17 or later (tested up to version 1.27)
+ Kubernetes version 1.17 or later (tested up to version 1.27)
## Architecture
![\[Architecture diagram of running stateful workloads with persistent data storage by using Amazon EFS\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/2487e285-269b-415b-a270-877f973e3aaf/images/ec8de63c-3307-4010-9e03-2bd7b9881fff.png)
The target architecture is comprised of the following infrastructure:
+ A virtual private cloud (VPC)
+ Two Availability Zones
+ A public subnet with a NAT gateway that provides internet access
+ A private subnet with an Amazon EKS cluster and Amazon EFS mount targets (also known as *mount points*)
+ Amazon EFS at the VPC level
The following is the environment infrastructure for the Amazon EKS cluster:
+ AWS Fargate profiles that accommodate the Kubernetes constructs at the namespace level
+ A Kubernetes namespace with:
+ Two application pods distributed across Availability Zones
+ One persistent volume claim (PVC) bound to a persistent volume (PV) at the cluster level
+ A cluster-wide PV that is bound to the PVC in the namespace and that points to the Amazon EFS mount targets in the private subnet, outside of the cluster
## Tools
**AWS services**
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that you can use to interact with AWS services from the command line.
+ [Amazon Elastic File System (Amazon EFS)](https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html) helps you create and configure shared file systems in the AWS Cloud. In this pattern, it provides a simple, scalable, fully managed, and shared file system for use with Amazon EKS.
+ [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) helps you run Kubernetes on AWS without needing to install or operate your own clusters.
+ [AWS Fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html) is a serverless compute engine for Amazon EKS. It creates and manages compute resources for your Kubernetes applications.
+ [AWS Key Management Service (AWS KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html) helps you create and control cryptographic keys to help protect your data.
**Other tools**
+ [Docker](https://www.docker.com/) is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers.
+ [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) is a command-line utility for creating and managing Kubernetes clusters on Amazon EKS.
+ [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) is a command-line interface that helps you run commands against Kubernetes clusters.
+ [jq](https://stedolan.github.io/jq/download/) is a command-line tool for parsing JSON.
**Code**
The code for this pattern is provided in the GitHub [Persistence Configuration with Amazon EFS on Amazon EKS using AWS Fargate](https://github.com/aws-samples/eks-efs-share-within-fargate) repo. The scripts are organized by epic, in the folders `epic01` through `epic06`, corresponding to the order in the [Epics](#run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate-epics) section in this pattern.
## Best practices
The target architecture includes the following services and components, and it follows [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected/) best practices:
+ Amazon EFS, which provides a simple, scalable, fully managed elastic NFS file system. This is used as a shared file system among all replications of the PoC application that are running in pods, which are distributed in the private subnets of the chosen Amazon EKS cluster.
+ An Amazon EFS mount target for each private subnet. This provides redundancy per Availability Zone within the virtual private cloud (VPC) of the cluster.
+ Amazon EKS, which runs the Kubernetes workloads. You must provision an Amazon EKS cluster before you use this pattern, as described in the [Prerequisites](#run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate-prereqs) section.
+ AWS KMS, which provides encryption at rest for the content that’s stored in the Amazon EFS file system.
+ Fargate, which manages the compute resources for the containers so that you can focus on business requirements instead of infrastructure burden. The Fargate profile is created for all private subnets. It provides redundancy per Availability Zone within the virtual private cloud (VPC) of the cluster.
+ Kubernetes Pods, for validating that content can be shared, consumed, and written by different instances of an application.
## Epics
### Provision an Amazon EKS cluster (optional)
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon EKS cluster. | If you already have a cluster deployed, skip to the next epic. Create an Amazon EKS cluster in your existing AWS account. In the [GitHub directory](https://github.com/aws-samples/eks-efs-share-within-fargate/tree/master/bootstrap), use one of the patterns to deploy an Amazon EKS cluster by using Terraform or eksctl. For more information, see [Creating an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) in the Amazon EKS documentation. In the Terraform pattern, there are also examples showing how to: link Fargate profiles to your Amazon EKS cluster, create an Amazon EFS file system, and deploy Amazon EFS CSI driver in your Amazon EKS cluster. | AWS administrator, Terraform or eksctl administrator, Kubernetes administrator |
| Export environment variables. | Run the env.sh script. This provides the information required in the next steps.source ./scripts/env.sh
Inform the AWS Account ID:
<13-digit-account-id>
Inform your AWS Region:
Inform your Amazon EKS Cluster Name:
Inform the Amazon EFS Creation Token:
If not noted yet, you can get all the information requested above with the following CLI commands.# ACCOUNT ID
aws sts get-caller-identity --query "Account" --output text
# REGION CODE
aws configure get region
# CLUSTER EKS NAME
aws eks list-clusters --query "clusters" --output text
# GENERATE EFS TOKEN
uuidgen
| AWS systems administrator |
### Create a Kubernetes namespace and a linked Fargate profile
| Task | Description | Skills required |
| --- | --- | --- |
| Create a Kubernetes namespace and Fargate profile for application workloads. | Create a namespace for receiving the application workloads that interact with Amazon EFS. Run the `create-k8s-ns-and-linked-fargate-profile.sh` script. You can choose to use a custom namespace name or the default provided namespace `poc-efs-eks-fargate`.**With a custom application namespace name:**export $APP_NAMESPACE=
./scripts/epic01/create-k8s-ns-and-linked-fargate-profile.sh \
-c "$CLUSTER_NAME" -n "$APP_NAMESPACE"
**Without a custom application namespace name:**./scripts/epic01/create-k8s-ns-and-linked-fargate-profile.sh \
-c "$CLUSTER_NAME"
where `$CLUSTER_NAME` is the name of your Amazon EKS cluster. The `-n ` parameter is optional; if not informed, a default generated namespace name will be provided. | Kubernetes user with granted permissions |
### Create an Amazon EFS file system
| Task | Description | Skills required |
| --- | --- | --- |
| Generate a unique token. | Amazon EFS requires a creation token to ensure idempotent operation (calling the operation with the same creation token has no effect). To meet this requirement, you must generate a unique token through an available technique. For example, you can generate a universally unique identifier (UUID) to use as a creation token. | AWS systems administrator |
| Create an Amazon EFS file system. | Create the file system for receiving the data files that are read and written by the application workloads. You can create an encrypted or non-encrypted file system. (As a best practice, the code for this pattern creates an encrypted system to enable encryption at rest by default.) You can use a unique, symmetric AWS KMS key to encrypt your file system. If a custom key is not specified, an AWS managed key is used.Use the create-efs.sh script to create an encrypted or non-encrypted Amazon EFS file system, after you generate a unique token for Amazon EFS.**With encryption at rest, without a KMS key:**./scripts/epic02/create-efs.sh \
-c "$CLUSTER_NAME" \
-t "$EFS_CREATION_TOKEN"
where `$CLUSTER_NAME` is the name of your Amazon EKS cluster and `$EFS_CREATION_TOKEN` is a unique creation token for the file system.**With encryption at rest, with a KMS key:**./scripts/epic02/create-efs.sh \
-c "$CLUSTER_NAME" \
-t "$EFS_CREATION_TOKEN" \
-k "$KMS_KEY_ALIAS"
where `$CLUSTER_NAME` is the name of your Amazon EKS cluster, `$EFS_CREATION_TOKEN` is a unique creation token for the file system, and `$KMS_KEY_ALIAS` is the alias for the KMS key.**Without encryption:**./scripts/epic02/create-efs.sh -d \
-c "$CLUSTER_NAME" \
-t "$EFS_CREATION_TOKEN"
where `$CLUSTER_NAME` is the name of your Amazon EKS cluster, `$EFS_CREATION_TOKEN` is a unique creation token for the file system, and `–d` disables encryption at rest. | AWS systems administrator |
| Create a security group. | Create a security group to allow the Amazon EKS cluster to access the Amazon EFS file system. | AWS systems administrator |
| Update the inbound rule for the security group. | Update the inbound rules of the security group to allow incoming traffic for the following settings:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate.html) | AWS systems administrator |
| Add a mount target for each private subnet. | For each private subnet of the Kubernetes cluster, create a mount target for the file system and the security group. | AWS systems administrator |
### Install Amazon EFS components into the Kubernetes cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the Amazon EFS CSI driver. | Deploy the Amazon EFS CSI driver into the cluster. The driver provisions storage according to persistent volume claims created by applications. Run the `create-k8s-efs-csi-sc.sh` script to deploy the Amazon EFS CSI driver and the storage class into the cluster../scripts/epic03/create-k8s-efs-csi-sc.sh
This script uses the `kubectl` utility, so make sure that the context has been configured and point to the desired Amazon EKS cluster. | Kubernetes user with granted permissions |
| Deploy the storage class. | Deploy the storage class into the cluster for the Amazon EFS provisioner (efs.csi.aws.com). | Kubernetes user with granted permissions |
### Install the PoC application into the Kubernetes cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the persistent volume. | Deploy the persistent volume, and link it to the created storage class and to the ID of the Amazon EFS file system. The application uses the persistent volume to read and write content. You can specify any size for the persistent volume in the storage field. Kubernetes requires this field, but because Amazon EFS is an elastic file system, it does not enforce any file system capacity. You can deploy the persistent volume with or without encryption. (The Amazon EFS CSI driver enables encryption by default, as a best practice.) Run the `deploy-poc-app.sh` script to deploy the persistent volume, the persistent volume claim, and the two workloads.**With encryption in transit:**./scripts/epic04/deploy-poc-app.sh \
-t "$EFS_CREATION_TOKEN"
where `$EFS_CREATION_TOKEN` is the unique creation token for the file system.**Without encryption in transit:**./scripts/epic04/deploy-poc-app.sh -d \
-t "$EFS_CREATION_TOKEN"
where `$EFS_CREATION_TOKEN` is the unique creation token for the file system, and `–d` disables encryption in transit. | Kubernetes user with granted permissions |
| Deploy the persistent volume claim requested by the application. | Deploy the persistent volume claim requested by the application, and link it to the storage class. Use the same access mode as the persistent volume you created previously. You can specify any size for the persistent volume claim in the storage field. Kubernetes requires this field, but because Amazon EFS is an elastic file system, it does not enforce any file system capacity. | Kubernetes user with granted permissions |
| Deploy workload 1. | Deploy the pod that represents workload 1 of the application. This workload writes content to the file `/data/out1.txt`. | Kubernetes user with granted permissions |
| Deploy workload 2. | Deploy the pod that represents workload 2 of the application. This workload writes content to the file `/data/out2.txt`. | Kubernetes user with granted permissions |
### Validate file system persistence, durability, and shareability
| Task | Description | Skills required |
| --- | --- | --- |
| Check the status of the `PersistentVolume`. | Enter the following command to check the status of the `PersistentVolume`.kubectl get pv
For an example output, see the [Additional information](#run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate-additional) section. | Kubernetes user with granted permissions |
| Check the status of the `PersistentVolumeClaim`. | Enter the following command to check the status of the `PersistentVolumeClaim`.kubectl -n poc-efs-eks-fargate get pvc
For an example output, see the [Additional information](#run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate-additional) section. | Kubernetes user with granted permissions |
| Validate that workload 1 can write to the file system. | Enter the following command to validate that workload 1 is writing to `/data/out1.txt`.kubectl exec -ti poc-app1 -n poc-efs-eks-fargate -- tail -f /data/out1.txt
The results are similar to the following:...
Thu Sep 3 15:25:07 UTC 2023 - PoC APP 1
Thu Sep 3 15:25:12 UTC 2023 - PoC APP 1
Thu Sep 3 15:25:17 UTC 2023 - PoC APP 1
...
| Kubernetes user with granted permissions |
| Validate that workload 2 can write to the file system. | Enter the following command to validate that workload 2 is writing to `/data/out2.txt`.kubectl -n $APP_NAMESPACE exec -ti poc-app2 -- tail -f /data/out2.txt
The results are similar to the following:...
Thu Sep 3 15:26:48 UTC 2023 - PoC APP 2
Thu Sep 3 15:26:53 UTC 2023 - PoC APP 2
Thu Sep 3 15:26:58 UTC 2023 - PoC APP 2
...
| Kubernetes user with granted permissions |
| Validate that workload 1 can read the file written by workload 2. | Enter the following command to validate that workload 1 can read the `/data/out2.txt` file written by workload 2.kubectl exec -ti poc-app1 -n poc-efs-eks-fargate -- tail -n 3 /data/out2.txt
The results are similar to the following:...
Thu Sep 3 15:26:48 UTC 2023 - PoC APP 2
Thu Sep 3 15:26:53 UTC 2023 - PoC APP 2
Thu Sep 3 15:26:58 UTC 2023 - PoC APP 2
...
| Kubernetes user with granted permissions |
| Validate that workload 2 can read the file written by workload 1. | Enter the following command to validate that workload 2 can read the `/data/out1.txt` file written by workload 1.kubectl -n $APP_NAMESPACE exec -ti poc-app2 -- tail -n 3 /data/out1.txt
The results are similar to the following:...
Thu Sep 3 15:29:22 UTC 2023 - PoC APP 1
Thu Sep 3 15:29:27 UTC 2023 - PoC APP 1
Thu Sep 3 15:29:32 UTC 2023 - PoC APP 1
...
| Kubernetes user with granted permissions |
| Validate that files are retained after you remove application components. | Next, you use a script to remove the application components (persistent volume, persistent volume claim, and pods), and validate that the files `/data/out1.txt` and `/data/out2.txt` are retained in the file system. Run the `validate-efs-content.sh` script by using the following command../scripts/epic05/validate-efs-content.sh \
-t "$EFS_CREATION_TOKEN"
where `$EFS_CREATION_TOKEN` is the unique creation token for the file system.The results are similar to the following:pod/poc-app-validation created
Waiting for pod get Running state...
Waiting for pod get Running state...
Waiting for pod get Running state...
Results from execution of 'find /data' on validation process pod:
/data
/data/out2.txt
/data/out1.txt
| Kubernetes user with granted permissions, System administrator |
### Monitor operations
| Task | Description | Skills required |
| --- | --- | --- |
| Monitor application logs. | As part of a day-two operation, ship the application logs to Amazon CloudWatch for monitoring. | AWS systems administrator, Kubernetes user with granted permissions |
| Monitor Amazon EKS and Kubernetes containers with Container Insights. | As part of a day-two operation, monitor the Amazon EKS and Kubernetes systems by using Amazon CloudWatch Container Insights. This tool collects, aggregates, and summarizes metrics from containerized applications at different levels and dimensions. For more information, see the [Related resources](#run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate-resources) section. | AWS systems administrator, Kubernetes user with granted permissions |
| Monitor Amazon EFS with CloudWatch. | As part of a day-two operation, monitor the file systems using Amazon CloudWatch, which collects and processes raw data from Amazon EFS into readable, near real-time metrics. For more information, see the [Related resources](#run-stateful-workloads-with-persistent-data-storage-by-using-amazon-efs-on-amazon-eks-with-aws-fargate-resources) section. | AWS systems administrator |
### Clean up resources
| Task | Description | Skills required |
| --- | --- | --- |
| Clean up all created resources for the pattern. | After you complete this pattern, clean up all resources, to avoid incurring AWS charges. Run the `clean-up-resources.sh` script to remove all resources after you have finished using the PoC application. Complete one of the following options.**With encryption at rest, with a KMS key:**./scripts/epic06/clean-up-resources.sh \
-c "$CLUSTER_NAME" \
-t "$EFS_CREATION_TOKEN" \
-k "$KMS_KEY_ALIAS"
where `$CLUSTER_NAME` is the name of your Amazon EKS cluster, `$EFS_CREATION_TOKEN` is the creation token for the file system, and `$KMS_KEY_ALIAS` is the alias for the KMS key.**Without encryption at rest:**./scripts/epic06/clean-up-resources.sh \
-c "$CLUSTER_NAME" \
-t "$EFS_CREATION_TOKEN"
where `$CLUSTER_NAME` is the name of your Amazon EKS cluster and `$EFS_CREATION_TOKEN` is the creation token for the file system. | Kubernetes user with granted permissions, System administrator |
## Related resources
**References**
+ [AWS Fargate for Amazon EKS now supports Amazon EFS](https://aws.amazon.com/blogs/aws/new-aws-fargate-for-amazon-eks-now-supports-amazon-efs/) (announcement)
+ [How to capture application logs when using Amazon EKS on AWS Fargate](https://aws.amazon.com/blogs/containers/how-to-capture-application-logs-when-using-amazon-eks-on-aws-fargate/) (blog post)
+ [Using Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html) (Amazon CloudWatch documentation)
+ [Setting Up Container Insights on Amazon EKS and Kubernetes](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) (Amazon CloudWatch documentation)
+ [Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html) (Amazon CloudWatch documentation)
+ [Monitoring Amazon EFS with Amazon CloudWatch](https://docs.aws.amazon.com/efs/latest/ug/monitoring-cloudwatch.html) (Amazon EFS documentation)
**GitHub tutorials and examples**
+ [Static provisioning](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/static_provisioning/README.md)
+ [Encryption in transit](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/encryption_in_transit/README.md)
+ [Accessing the file system from multiple pods](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/multiple_pods/README.md)
+ [Consuming Amazon EFS in StatefulSets](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/statefulset/README.md)
+ [Mounting subpaths](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/volume_path/README.md)
+ [Using Amazon EFS access points](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/access_points/README.md)
+ [Amazon EKS Blueprints for Terraform](https://aws-ia.github.io/terraform-aws-eks-blueprints/)
**Required tools**
+ [Installing the AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
+ [Installing eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)
+ [Installing kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html)
+ [Installing jq](https://stedolan.github.io/jq/download/)
## Additional information
The following is an example output of the `kubectl get pv` command.
```
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
poc-app-pv 1Mi RWX Retain Bound poc-efs-eks-fargate/poc-app-pvc efs-sc 3m56s
```
The following is an example output of the `kubectl -n poc-efs-eks-fargate get pvc` command.
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
poc-app-pvc Bound poc-app-pv 1Mi RWX efs-sc 4m34s
```
# Set up event-driven auto scaling in Amazon EKS by using Amazon EKS Pod Identity and KEDA
*Dipen Desai, Abhay Diwan, Kamal Joshi, and Mahendra Revanasiddappa, Amazon Web Services*
## Summary
Orchestration platforms, such as [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html), have streamlined the lifecycle management of container-based applications. This helps organizations focus on building, securing, operating, and maintaining container-based applications. As event-driven deployments become more common, organizations are more frequently scaling Kubernetes deployments based on various event sources. This method, combined with auto scaling, can result in significant cost savings by providing on-demand compute resources and efficient scaling that is tailored to application logic.
[KEDA](https://keda.sh/) is a Kubernetes-based event-driven autoscaler. KEDA helps you scale any container in Kubernetes based on the number of events that need to be processed. It is lightweight and integrates with any Kubernetes cluster. It also works with standard Kubernetes components, such as [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). KEDA also offers [TriggerAuthentication](https://keda.sh/docs/2.14/concepts/authentication/#re-use-credentials-and-delegate-auth-with-triggerauthentication), which is a feature that helps you delegate authentication. It allows you to describe authentication parameters that are separate from the ScaledObject and the deployment containers.
AWS provides AWS Identity and Access Management (IAM) roles that support diverse Kubernetes deployment options, including Amazon EKS, Amazon EKS Anywhere, Red Hat OpenShift Service on AWS (ROSA), and self-managed Kubernetes clusters on Amazon Elastic Compute Cloud (Amazon EC2). These roles use IAM constructs, such as OpenID Connect (OIDC) identity providers and IAM trust policies, to operate across different environments without relying directly on Amazon EKS services or APIs. For more information, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) in the Amazon EKS documentation.
[Amazon EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) simplifies the process for Kubernetes service accounts to assume IAM roles without requiring OIDC providers. It provides the ability to manage credentials for your applications. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance’s role, you associate an IAM role with a Kubernetes service account and configure your Pods to use the service account. This helps you use an IAM role across multiple clusters and simplifies policy management by enabling the reuse of permission policies across IAM roles.
By implementing KEDA with Amazon EKS Pod Identity, businesses can achieve efficient event-driven auto scaling and simplified credential management. Applications scale based on demand, which optimizes resource utilization and reduces costs.
This pattern helps you integrate Amazon EKS Pod Identity with KEDA. It showcases how you can use the `keda-operator` service account and delegate authentication with `TriggerAuthentication`. It also describes how to set up a trust relationship between an IAM role for the KEDA operator and an IAM role for the application. This trust relationship allows KEDA to monitor messages in the event queues and adjust scaling for the destination Kubernetes objects.
## Prerequisites and limitations
**Prerequisites**
+ AWS Command Line Interface (AWS CLI) version 2.13.17 or later, [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
+ Python version 3.11.5 or later, [installed](https://www.python.org/downloads/)
+ AWS SDK for Python (Boto3) version 1.34.135 or later, [installed](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html)
+ Helm version 3.12.3 or later, [installed](https://helm.sh/docs/intro/install/)
+ kubectl version 1.25.1 or later, [installed](https://kubernetes.io/docs/tasks/tools/)
+ Docker Engine version 26.1.1 or later, [installed](https://docs.docker.com/engine/install/)
+ An Amazon EKS cluster version 1.24 or later, [created](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html)
+ Prerequisites for creating the Amazon EKS Pod Identity agent, [met](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html#pod-id-agent-add-on-create)
**Limitations**
+ It is required that you establish a trust relationship between the `keda-operator` role and the `keda-identity` role. Instructions are provided in the [Epics](#event-driven-auto-scaling-with-eks-pod-identity-and-keda-epics) section of this pattern.
## Architecture
In this pattern, you create the following AWS resources:
+ **Amazon Elastic Container Registry (Amazon ECR) repository** – In this pattern, this repo is named `keda-pod-identity-registry`. This private repo is used to store Docker images of the sample application.
+ **Amazon Simple Queue Service (Amazon SQS) queue** – In this pattern, this queue is named `event-messages-queue`. The queue acts as a message buffer that collects and stores incoming messages. KEDA monitors the queue metrics, such as message count or queue length, and it automatically scales the application based on these metrics.
+ **IAM role for the application** – In this pattern, this role is named `keda-identity`. The `keda-operator` role assumes this role. This role allows access to the Amazon SQS queue.
+ **IAM role for the KEDA operator** – In this pattern, this role is named `keda-operator`. The KEDA operator uses this role to make the required AWS API calls. This role has permissions to assume the `keda-identity` role. Because of the trust relationship between the `keda-operator` and the `keda-identity` roles, the `keda-operator` role has Amazon SQS permissions.
Through the `TriggerAuthentication` and `ScaledObject` Kubernetes custom resources, the operator uses the `keda-identity` role to connect with an Amazon SQS queue. Based on the queue size, KEDA automatically scales the application deployment. It adds 1 pod for every 5 unread messages in the queue. In the default configuration, if there are no unread messages in the Amazon SQS queue, the application scales down to 0 pods. The KEDA operator monitors the queue at an interval that you specify.
The following image shows how you use Amazon EKS Pod Identity to provide the `keda-operator` role with secure access to the Amazon SQS queue.
![\[Using KEDA and Amazon EKS Pod Identity to automatically scale a Kubernetes-based application.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/56f7506d-e8d3-43e5-bec6-42267fedd0ae/images/05bdbd09-9eb8-4c0b-8c0d-efe38aecb683.png)
The diagram shows the following workflow:
1. You install the Amazon EKS Pod Identity agent in the Amazon EKS cluster.
1. You deploy KEDA operator in the KEDA namespace in the Amazon EKS cluster.
1. You create the `keda-operator` and `keda-identity` IAM roles in the target AWS account.
1. You establish a trust relationship between the IAM roles.
1. You deploy the application in the `security` namespace.
1. The KEDA operator polls messages in an Amazon SQS queue.
1. KEDA initiates HPA, which automatically scales the application based on the queue size.
## Tools
**AWS services**
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable.
+ [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.
+ [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
+ [Amazon Simple Queue Service (Amazon SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) provides a secure, durable, and available hosted queue that helps you integrate and decouple distributed software systems and components.
**Other tools**
+ [KEDA](https://keda.sh/) is a Kubernetes-based event-driven autoscaler.
**Code repository**
The code for this pattern is available in the GitHub [Event-driven auto scaling using EKS Pod Identity and KEDA](https://github.com/aws-samples/event-driven-autoscaling-using-podidentity-and-keda/tree/main) repository.
## Best practices
We recommend that you adhere to the following best practices:
+ [Amazon EKS best practices](https://docs.aws.amazon.com/eks/latest/best-practices/introduction.html)
+ [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html)
+ [Amazon SQS best practices](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-best-practices.html)
## Epics
### Create AWS resources
| Task | Description | Skills required |
| --- | --- | --- |
| Create the IAM role for the KEDA operator. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | AWS administrator |
| Create the IAM role for the sample application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | AWS administrator |
| Create an Amazon SQS queue. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | General AWS |
| Create an Amazon ECR repository. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | General AWS |
### Set up the Amazon EKS cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Deploy the Amazon EKS Pod Identity agent. | For the target Amazon EKS cluster, set up the Amazon EKS Pod Identity agent. Follow the instructions in [Set up the Amazon EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html#pod-id-agent-add-on-create) in the Amazon EKS documentation. | AWS DevOps |
| Deploy KEDA. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
| Assign the IAM role to the Kubernetes service account. | Follow the instructions in [Assign an IAM role to a Kubernetes service account](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-association.html) in the Amazon EKS documentation. Use the following values:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | AWS DevOps |
| Create a namespace. | Enter the following command to create a `security` namespace in the target Amazon EKS cluster:kubectl create ns security
| DevOps engineer |
### Deploy the sample application
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the application files. | Enter the following command to clone the [Event-driven auto scaling using EKS Pod Identity and KEDA repository](https://github.com/aws-samples/event-driven-autoscaling-using-podidentity-and-keda/tree/main) from GitHub:git clone https://github.com/aws-samples/event-driven-autoscaling-using-podidentity-and-keda.git
| DevOps engineer |
| Build the Docker image. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
| Push the Docker image to Amazon ECR. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html)You can find push commands by navigating to the Amazon ECR repository page and then choosing **View push commands**. | DevOps engineer |
| Deploy the sample application. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
| Assign the IAM role to the application service account. | Do one of the following to associate the `keda-identity` IAM role with the service account for the sample application:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
| Deploy `ScaledObject` and `TriggerAuthentication`. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
### Test auto scaling
| Task | Description | Skills required |
| --- | --- | --- |
| Send messages to the Amazon SQS queue. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
| Monitor the application pods. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) | DevOps engineer |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The KEDA operator cannot scale the application. | Enter the following command to check the logs of the `keda-operator` IAM role:kubectl logs -n keda -l app=keda-operator -c keda-operator
If there is an `HTTP 403` response code, then the application and the KEDA scaler do not have sufficient permissions to access the Amazon SQS queue. Complete the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html)If there is an `Assume-Role` error, then an [Amazon EKS node IAM role](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) is unable to assume the IAM role that is defined for `TriggerAuthentication`. Complete the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/event-driven-auto-scaling-with-eks-pod-identity-and-keda.html) |
## Related resources
+ [Set up the Amazon EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html) (Amazon EKS documentation)
+ [Deploying KEDA](https://keda.sh/docs/2.14/deploy/) (KEDA documentation)
+ [ScaledObject specification](https://keda.sh/docs/2.16/reference/scaledobject-spec/) (KEDA documentation)
+ [Authentication with TriggerAuthentication](https://keda.sh/docs/2.14/concepts/authentication/) (KEDA documentation)
# Streamline PostgreSQL deployments on Amazon EKS by using PGO
*Shalaka Dengale, Amazon Web Services*
## Summary
This pattern integrates the Postgres Operator from Crunchy Data (PGO) with Amazon Elastic Kubernetes Service (Amazon EKS) to streamline PostgreSQL deployments in cloud-native environments. PGO provides automation and scalability for managing PostgreSQL databases in Kubernetes. When you combine PGO with Amazon EKS, it forms a robust platform for deploying, managing, and scaling PostgreSQL databases efficiently.
This integration provides the following key benefits:
+ Automated deployment: Simplifies PostgreSQL cluster deployment and management.
+ Custom resource definitions (CRDs):** **Uses Kubernetes primitives for PostgreSQL management.
+ High availability: Supports automatic failover and synchronous replication.
+ Automated backups and restores:** **Streamlines backup and restore processes.
+ Horizontal scaling:** **Enables dynamic scaling of PostgreSQL clusters.
+ Version upgrades: Facilitates rolling upgrades with minimal downtime.
+ Security: Enforces encryption, access controls, and authentication mechanisms.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account.
+ [AWS Command Line Interface (AWS CLI) version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html), installed and configured on Linux, macOS, or Windows.
+ [AWS CLI Config](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html), to connect AWS resources from the command line.
+ [eksctl](https://github.com/eksctl-io/eksctl#installation), installed and configured on Linux, macOS, or Windows.
+ `kubectl`, installed and configured to access resources on your Amazon EKS cluster. For more information, see [Set up kubectl and eksctl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) in the Amazon EKS documentation.
+ Your computer terminal configured to access the Amazon EKS cluster. For more information, see [Configure your computer to communicate with your cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#eks-configure-kubectl) in the Amazon EKS documentation.
**Product versions**
+ Kubernetes versions 1.21–1.24 or later (see the [PGO documentation](https://access.crunchydata.com/documentation/postgres-operator/5.2.5/)).
+ PostgreSQL version 10 or later. This pattern uses PostgreSQL version 16.
**Limitations**
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see the [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) page, and choose the link for the service.
## Architecture
**Target technology stack **
+ Amazon EKS
+ Amazon Virtual Private Cloud (Amazon VPC)
+ Amazon Elastic Compute Cloud (Amazon EC2)
**Target architecture **
![\[Architecture for using PGO with three Availability Zones and two replicas, PgBouncer, and PGO operator.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/4c164012-7527-4ebe-b6a7-c129600328d6/images/26a5572b-405b-4634-b96a-91254c3ea2c1.png)
This pattern builds an architecture that contains an Amazon EKS cluster with three nodes. Each node runs on a set of EC2 instances in the backend. This PostgreSQL setup follows a primary replica architecture, which is particularly effective for read-heavy use cases. The architecture includes the following components:
+ **Primary database container (pg-primary)** hosts the main PostgreSQL instance where all write operations are directed.
+ **Secondary replica containers (pg-replica)** host the PostgreSQL instances that replicate the data from the primary database and handle read operations.
+ **PgBouncer** is a lightweight connection pooler for PostgreSQL databases that's included with PGO. It sits between the client and the PostgreSQL server, and acts as an intermediary for database connections.
+ **PGO** automates the deployment and management of PostgreSQL clusters in this Kubernetes environment.
+ **Patroni** is an open-source tool that manages and automates high availability configurations for PostgreSQL. It's included with PGO. When you use Patroni with PGO in Kubernetes, it plays a crucial role in ensuring the resilience and fault tolerance of a PostgreSQL cluster. For more information, see the [Patroni documentation](https://patroni.readthedocs.io/en/latest/).
The workflow includes these steps:
+ **Deploy the PGO operator**. You deploy the PGO operator on your Kubernetes cluster that runs on Amazon EKS. This can be done by using Kubernetes manifests or Helm charts. This pattern uses Kubernetes manifests.
+ **Define PostgreSQL instances**. When the operator is running, you create custom resources (CRs) to specify the desired state of PostgreSQL instances. This includes configurations such as storage, replication, and high availability settings.
+ **Operator management**. You interact with the operator through Kubernetes API objects such as CRs to create, update, or delete PostgreSQL instances.
+ **Monitoring and maintenance**. You can monitor the health and performance of the PostgreSQL instances running on Amazon EKS. Operators often provide metrics and logging for monitoring purposes. You can perform routine maintenance tasks such as upgrades and patching as necessary. For more information, see [Monitor your cluster performance and view logs](https://docs.aws.amazon.com/eks/latest/userguide/eks-observe.html) in the Amazon EKS documentation.
+ **Scaling and backup**: You can use the features provided by the operator to scale PostgreSQL instances and manage backups.
This pattern doesn't cover monitoring, maintenance, and backup operations.
**Automation and scale**
+ You can use CloudFormation to automate the infrastructure creation. For more information, see [Create Amazon EKS resources with CloudFormation](https://docs.aws.amazon.com/eks/latest/userguide/creating-resources-with-cloudformation.html) in the Amazon EKS documentation.
+ You can use GitVersion or Jenkins build numbers to automate the deployment of database instances.
## Tools
**AWS services**
+ [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.
+ [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) is an open-source tool that helps you interact with AWS services through commands in your command line shell.
**Other tools**
+ [eksctl](https://eksctl.io/) is a simple command line tool for creating clusters on Amazon EKS.
+ [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) is a command line utility for running commands against Kubernetes clusters.
+ [PGO](https://github.com/CrunchyData/postgres-operator) automates and scales the management of PostgreSQL databases in Kubernetes.
## Best practices
Follow these best practices to ensure a smooth and efficient deployment:
+ **Secure your EKS cluster**. Implement security best practices for your EKS cluster, such as using AWS Identity and Access Management (IAM) roles for service accounts (IRSA), network policies, and VPC security groups. Limit access to the EKS cluster API server, and encrypt communications between nodes and the API server by using TLS.
+ **Ensure version compatibility** between PGO and Kubernetes running on Amazon EKS. Some PGO features might require specific Kubernetes versions or introduce compatibility limitations. For more information, see [Components and Compatibility](https://access.crunchydata.com/documentation/postgres-operator/5.2.5/references/components/) in the PGO documentation.
+ **Plan resource allocation** for your PGO deployment, including CPU, memory, and storage. Consider the resource requirements of both PGO and the PostgreSQL instances it manages. Monitor resource usage and scale resources as needed.
+ **Design for high availability**. Design your PGO deployment for high availability to minimize downtime and ensure reliability. Deploy multiple replicas of PGO across multiple Availability Zones for fault tolerance.
+ **Implement backup and restore procedures** for your PostgreSQL databases that PGO manages. Use features provided by PGO or third-party backup solutions that are compatible with Kubernetes and Amazon EKS.
+ **Set up monitoring and logging** for your PGO deployment to track performance, health, and events. Use tools such as Prometheus for monitoring metrics and Grafana for visualization. Configure logging to capture PGO logs for troubleshooting and auditing.
+ **Configure networking** properly to allow communications between PGO, PostgreSQL instances, and other services in your Kubernetes cluster. Use Amazon VPC networking features and Kubernetes networking plugins such as Calico or [Amazon VPC CNI](https://github.com/aws/amazon-vpc-cni-k8s) for network policy enforcement and traffic isolation.
+ **Choose appropriate storage options** for your PostgreSQL databases, considering factors such as performance, durability, and scalability. Use Amazon Elastic Block Store (Amazon EBS) volumes or AWS managed storage services for persistent storage. For more information, see [Store Kubernetes volumes with Amazon EBS](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) in the Amazon EKS documentation.
+ **Use infrastructure as code (IaC) tools** such as CloudFormation to automate the deployment and configuration of PGO on Amazon EKS. Define infrastructure components—including the EKS cluster, networking, and PGO resources—as code for consistency, repeatability, and version control.
## Epics
### Create an IAM role
| Task | Description | Skills required |
| --- | --- | --- |
| Create an IAM role. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) | AWS administrator |
### Create an Amazon EKS cluster
| Task | Description | Skills required |
| --- | --- | --- |
| Create an Amazon EKS cluster. | If you've already deployed a cluster, skip this step. Otherwise, deploy an Amazon EKS cluster in your current AWS account by using `eksctl`, Terraform, or CloudFormation. This pattern uses `eksctl` for cluster deployment.This pattern uses Amazon EC2 as a node group for Amazon EKS. If you want to use AWS Fargate, see the `managedNodeGroups` configuration in the [eksctl documentation](https://eksctl.io/usage/schema/#managedNodeGroups).[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) | AWS administrator, Terraform or eksctl administrator, Kubernetes administrator |
| Validate the status of the cluster. | Run the following command to see the current status of nodes in the cluster:kubectl get nodes
If you encounter errors, see the [troubleshooting section](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html) of the Amazon EKS documentation. | AWS administrator, Terraform or eksctl administrator, Kubernetes administrator |
### Create an OIDC identity provider
| Task | Description | Skills required |
| --- | --- | --- |
| Enable the IAM OIDC provider. | As a prerequisite for the Amazon EBS Container Storage Interface (CSI) driver, you must have an existing IAM OpenID Connect (OIDC) provider for your cluster.Enable the IAM OIDC provider by using the following command:eksctl utils associate-iam-oidc-provider --region={region} --cluster={YourClusterNameHere} --approveFor more information about this step, see the [Amazon EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html). | AWS administrator |
| Create an IAM role for the Amazon EBS CSI driver. | Use the following `eksctl` command to create the IAM role for the CSI driver:eksctl create iamserviceaccount \
--region {RegionName} \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster {YourClusterNameHere} \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve \
--role-only \
--role-name AmazonEKS_EBS_CSI_DriverRole
If you use encrypted Amazon EBS drives, you have to configure the policy further. For instructions, see the [Amazon EBS SCI driver documentation](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md#installation-1). | AWS administrator |
| Add the Amazon EBS CSI driver. | Use the following `eksctl` command to add the Amazon EBS CSI driver:eksctl create addon \
--name aws-ebs-csi-driver \
--cluster service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity \
--query Account \
--output text):role/AmazonEKS_EBS_CSI_DriverRole \
--force
| AWS administrator |
### Install PGO
| Task | Description | Skills required |
| --- | --- | --- |
| Clone the PGO repository. | Clone the GitHub repository for PGO:git clone https://github.com/CrunchyData/postgres-operator-examples.git
| AWS DevOps |
| Provide the role details for service account creation. | To grant the Amazon EKS cluster access to the required AWS resources, specify the Amazon Resource Name (ARN) of the OIDC role that you created earlier in the `service_account.yaml` file that is located in [GitHub](https://github.com/CrunchyData/postgres-operator/blob/main/config/rbac/cluster/service_account.yaml).cd postgres-operator-examples
---
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:::role/ # Update the OIDC role ARN created earlier
| AWS administrator, Kubernetes administrator |
| Create the namespace and PGO prerequisites. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) | Kubernetes administrator |
| Verify the creation of pods. | Verify that the namespace and default configuration were created:kubectl get pods -n postgres-operator
| AWS administrator, Kubernetes administrator |
| Verify PVCs. | Use the following command to verify persistent volume claims (PVCs):kubectl describe pvc -n postgres-operator
| AWS administrator, Kubernetes administrator |
### Create and deploy an operator
| Task | Description | Skills required |
| --- | --- | --- |
| Create an operator. | Revise the contents of the file located at `/kustomize/postgres/postgres.yaml` to match the following:spec:
instances:
- name: pg-1
replicas: 3
patroni:
dynamicConfiguration:
postgresql:
pg_hba:
- "host all all 0.0.0.0/0 trust" # this line enabled logical replication with programmatic access
- "host all postgres 127.0.0.1/32 md5"
synchronous_mode: true
users:
- name: replicator
databases:
- testdb
options: "REPLICATION"
These updates do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) | AWS administrator, DBA, Kubernetes administrator |
| Deploy the operator. | Deploy the PGO operator to enable the streamlined management and operation of PostgreSQL databases in Kubernetes environments:kubectl apply -k kustomize/postgres
| AWS administrator, DBA, Kubernetes administrator |
| Verify the deployment. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html)From the command output, note the primary replica (`primary_pod_name`) and read replica (`read_pod_name`). You will uses these in the next steps. | AWS administrator, DBA, Kubernetes administrator |
### Verify streaming replication
| Task | Description | Skills required |
| --- | --- | --- |
| Write data to the primary replica. | Use the following commands to connect to the PostgreSQL primary replica and write data to the database:kubectl exec -it bash -n postgres-operator
psql
CREATE TABLE customers (firstname text, customer_id serial, date_created timestamp);
\dt
| AWS administrator, Kubernetes administrator |
| Confirm that the read replica has the same data. | Connect to the PostgreSQL read replica and check whether the streaming replication is working correctly:kubectl exec -it {read_pod_name} bash -n postgres-operatorpsql
\dt
The read replica should have the table that you created in the primary replica in the previous step. | AWS administrator, Kubernetes administrator |
## Troubleshooting
| Issue | Solution |
| --- | --- |
| The pod doesn’t start. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) |
| Replicas are significantly behind the primary database. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) |
| You don’t have visibility into the performance and health of the PostgreSQL cluster. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) |
| Replication doesn’t work. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/streamline-postgresql-deployments-amazon-eks-pgo.html) |
## Related resources
+ [Amazon Elastic Kubernetes Service](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/amazon-elastic-kubernetes-service.html) (*Overview of Deployment Options on AWS* whitepaper)
+ [CloudFormation](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/aws-cloudformation.html) (*Overview of Deployment Options on AWS* whitepaper)
+ [Get started with Amazon EKS – eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) (*Amazon EKS User Guide*)
+ [Set up kubectl and eksctl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) (*Amazon EKS User Guide*)
+ [Create a role for OpenID Connect federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html) (*IAM User Guide*)
+ [Configuring settings for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) (*AWS CLI User Guide*)
+ [Crunchy Postgres for Kubernetes documentation](https://access.crunchydata.com/documentation/postgres-operator/latest)
+ [Crunch & Learn: Crunchy Postgres for Kubernetes 5.0](https://www.youtube-nocookie.com/embed/IIf9WZO3K50) (video)
# Simplify application authentication with mutual TLS in Amazon ECS by using Application Load Balancer
*Olawale Olaleye and Shamanth Devagari, Amazon Web Services*
## Summary
This pattern helps you to simplify your application authentication and offload security burdens with mutual TLS in Amazon Elastic Container Service (Amazon ECS) by using [Application Load Balancer (ALB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/mutual-authentication.html). With ALB, you can authenticate X.509 client certificates from AWS Private Certificate Authority. This powerful combination helps to achieve secure communication between your services, reducing the need for complex authentication mechanisms within your applications. In addition, the pattern uses Amazon Elastic Container Registry (Amazon ECR) to store container images.
The example in this pattern uses Docker images from a public gallery to create the sample workloads initially. Subsequently, new Docker images are built to be stored in Amazon ECR. For the source, consider a Git-based system such as GitHub, GitLab, or Bitbucket, or use Amazon Simple Storage Service Amazon S3 (Amazon S3). For building the Docker images, consider using AWS CodeBuild for the subsequent images.
## Prerequisites and limitations
**Prerequisites**
+ An active AWS account with access to deploy AWS CloudFormation stacks. Make sure that you have AWS Identity and Access Management (IAM) [user or role permissions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/control-access-with-iam.html) to deploy CloudFormation.
+ AWS Command Line Interface (AWS CLI) [installed](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). [Configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) your AWS credentials on your local machine or in your environment by either using the AWS CLI or by setting the environment variables in the `~/.aws/credentials` file.
+ OpenSSL [installed](https://www.openssl.org/).
+ Docker [installed](https://www.docker.com/get-started/).
+ Familiarity with the AWS services described in [Tools](#simplify-application-authentication-with-mutual-tls-in-amazon-ecs-tools).
+ Knowledge of Docker and NGINX.
**Limitations**
+ Mutual TLS for Application Load Balancer only supports X.509v3 client certificates. X.509v1 client certificates are not supported.
+ The CloudFormation template that is provided in this pattern’s code repository doesn’t include provisioning a CodeBuild project as part of the stack.
+ Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service.
**Product versions**
+ Docker version 27.3.1 or later
+ AWS CLI version 2.14.5 or later
## Architecture
The following diagram shows the architecture components for this pattern.
![\[Workflow to authenticate with mutual TLS using Application Load Balancer.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a343fa4e-097f-416b-9c83-01a28eb57dc3/images/e1371297-b987-4487-9b13-8120933c921f.png)
The diagram shows the following workflow:
1. Create a Git repository, and commit the application code to the repository.
1. Create a private certificate authority (CA) in AWS Private CA.
1. Create a CodeBuild project. The CodeBuildproject is triggered by commit changes and creates the Docker image and publishes the built image to Amazon ECR.
1. Copy the certificate chain and certificate body from the CA, and upload the certificate bundle to Amazon S3.
1. Create a trust store with the CA bundle that you uploaded to Amazon S3. Associate the trust store with the mutual TLS listeners on the Application Load Balancer (ALB).
1. Use the private CA to issue client certificates for the container workloads. Also create a private TLS certificate using AWS Private CA.
1. Import the private TLS certificate into AWS Certificate Manager (ACM), and use it with the ALB.
1. The container workload in `ServiceTwo` uses the issued client certificate to authenticate with the ALB when it communicates with the container workload in `ServiceOne`.
1. The container workload in `ServiceOne` uses the issued client certificate to authenticate with the ALB when it communicates with the container workload in `ServiceTwo`.
**Automation and scale**
This pattern can be fully automated by using CloudFormation, AWS Cloud Development Kit (AWS CDK) , or API operations from an SDK to provision the AWS resources.
You can use AWS CodePipeline to implement a continuous integration and continuous deployment (CI/CD) pipeline using CodeBuild to automate container image build process and deploying new releases to the Amazon ECS cluster services.
## Tools
**AWS services **
+ [AWS Certificate Manager (ACM)](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html) helps you create, store, and renew public and private SSL/TLS X.509 certificates and keys that protect your AWS websites and applications.
+ [AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions.
+ [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html) is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.
+ [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable.
+ [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a highly scalable, fast container management service for running, stopping, and managing containers on a cluster. You can run your tasks and services on a serverless infrastructure that is managed by AWS Fargate. Alternatively, for more control over your infrastructure, you can run your tasks and services on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances that you manage.
+ [Amazon ECS Exec](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) allows you to directly interact with containers without needing to first interact with the host container operating system, open inbound ports, or manage SSH keys. You can use ECS Exec to run commands in, or get a shell to, a container running on an Amazon EC2 instance or on AWS Fargate.
+ [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. For example, you can distribute traffic across Amazon EC2 instances, containers, and IP addresses, in one or more Availability Zones. ELB monitors the health of its registered targets, and routes traffic only to the healthy targets. ELB scales your load balancer as your incoming traffic changes over time. It can automatically scale to the majority of workloads.
+ [AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html) helps you run containers without needing to manage servers or Amazon EC2 instances. Fargate is compatible with both Amazon ECS and Amazon Elastic Kubernetes Service (Amazon EKS). You can run your Amazon ECS tasks and services with the Fargate launch type or a Fargate capacity provider. To do so, package your application in containers, specify the CPU and memory requirements, define networking and IAM policies, and launch the application. Each Fargate task has its own isolation boundary and doesn’t share the underlying kernel, CPU resources, memory resources, or elastic network interface with another task.
+ [AWS Private Certificate Authority](https://docs.aws.amazon.com/privateca/latest/userguide/PcaWelcome.html) enables creation of private certificate authority (CA) hierarchies, including root and subordinate CAs, without the investment and maintenance costs of operating an on-premises CA.
**Other tools**** **
+ [Docker](https://www.docker.com/) is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers.
+ [GitHub](https://docs.github.com/en/repositories/creating-and-managing-repositories/quickstart-for-repositories), [GitLab](https://docs.gitlab.com/ee/user/get_started/get_started_projects.html), and [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/tutorial-learn-bitbucket-with-git/) are some of the commonly used Git-based source control system to keep track of source code changes.
+ [NGINX Open Source](https://nginx.org/en/docs/?_ga=2.187509224.1322712425.1699399865-405102969.1699399865) is an open source load balancer, content cache, and web server. This pattern uses it as a web server.
+ [OpenSSL](https://www.openssl.org/) is an open source library that provides services that are used by the OpenSSL implementations of TLS and CMS.
**Code repository**
The code for this pattern is available in the GitHub [mTLS-with-Application-Load-Balancer-in-Amazon-ECS](https://github.com/aws-samples/mTLS-with-Application-Load-Balancer-in-Amazon-ECS) repository.
## Best practices
+ Use Amazon ECS Exec to run commands or get a shell to a container running on Fargate. You can also use ECS Exec to help collect diagnostic information for debugging.
+ Use security groups and network access control lists (ACLs) to control inbound and outbound traffic between the services. Fargate tasks receive an IP address from the configured subnet in your virtual private cloud (VPC).
## Epics
### Create the repository
| Task | Description | Skills required |
| --- | --- | --- |
| Download the source code. | To download this pattern’s source code, fork or clone the GitHub [mTLS-with-Application-Load-Balancer-in-Amazon-ECS](https://github.com/aws-samples/mTLS-with-Application-Load-Balancer-in-Amazon-ECS) repository. | DevOps engineer |
| Create a Git repository. | To create a Git repository to contain the Dockerfile and the `buildspec.yaml` files, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html)`git clone https://github.com/aws-samples/mTLS-with-Application-Load-Balancer-in-Amazon-ECS.git` | DevOps engineer |
### Create CA and generate certificates
| Task | Description | Skills required |
| --- | --- | --- |
| Create a private CA in AWS Private CA. | To create a private certificate authority (CA), run the following commands in your terminal. Replace the values in the example variables with your own values. export AWS_DEFAULT_REGION="us-west-2"
export SERVICES_DOMAIN="www.example.com"
export ROOT_CA_ARN=`aws acm-pca create-certificate-authority \
--certificate-authority-type ROOT \
--certificate-authority-configuration \
"KeyAlgorithm=RSA_2048,
SigningAlgorithm=SHA256WITHRSA,
Subject={
Country=US,
State=WA,
Locality=Seattle,
Organization=Build on AWS,
OrganizationalUnit=mTLS Amazon ECS and ALB Example,
CommonName=${SERVICES_DOMAIN}}" \
--query CertificateAuthorityArn --output text`
For more details, see [Create a private CA in AWS Private CA](https://docs.aws.amazon.com/privateca/latest/userguide/create-CA.html) in the AWS documentation. | DevOps engineer, AWS DevOps |
| Create and install your private CA certificate. | To create and install a certificate for your private root CA, run the following commands in your terminal:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html) | AWS DevOps, DevOps engineer |
| Request a managed certificate. | To request a private certificate in AWS Certificate Manager to use with your private ALB, use the following command:export TLS_CERTIFICATE_ARN=`aws acm request-certificate \
--domain-name "*.${DOMAIN_DOMAIN}" \
--certificate-authority-arn ${ROOT_CA_ARN} \
--query CertificateArn --output text`
| DevOps engineer, AWS DevOps |
| Use the private CA to issue a client certificate. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html)`openssl req -out client_csr1.pem -new -newkey rsa:2048 -nodes -keyout client_private-key1.pem``openssl req -out client_csr2.pem -new -newkey rsa:2048 -nodes -keyout client_private-key2.pem`This command returns the CSR and the private key for the two services. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html)SERVICE_ONE_CERT_ARN=`aws acm-pca issue-certificate \
--certificate-authority-arn ${ROOT_CA_ARN} \
--csr fileb://client_csr1.pem \
--signing-algorithm "SHA256WITHRSA" \
--validity Value=5,Type="YEARS" --query CertificateArn --output text`
echo "SERVICE_ONE_CERT_ARN: ${SERVICE_ONE_CERT_ARN}"
aws acm-pca get-certificate \
--certificate-authority-arn ${ROOT_CA_ARN} \
--certificate-arn ${SERVICE_ONE_CERT_ARN} \
| jq -r '.Certificate' > client_cert1.cert
SERVICE_TWO_CERT_ARN=`aws acm-pca issue-certificate \
--certificate-authority-arn ${ROOT_CA_ARN} \
--csr fileb://client_csr2.pem \
--signing-algorithm "SHA256WITHRSA" \
--validity Value=5,Type="YEARS" --query CertificateArn --output text`
echo "SERVICE_TWO_CERT_ARN: ${SERVICE_TWO_CERT_ARN}"
aws acm-pca get-certificate \
--certificate-authority-arn ${ROOT_CA_ARN} \
--certificate-arn ${SERVICE_TWO_CERT_ARN} \
| jq -r '.Certificate' > client_cert2.cert
For more information, see [Issue private end-entity certificates](https://docs.aws.amazon.com/privateca/latest/userguide/PcaIssueCert.html) in the AWS documentation. | DevOps engineer, AWS DevOps |
### Provision AWS services
| Task | Description | Skills required |
| --- | --- | --- |
| Provision AWS services with the CloudFormation template. | To provision the virtual private cloud (VPC), Amazon ECS cluster, Amazon ECS services, Application Load Balancer, and Amazon Elastic Container Registry (Amazon ECR), use the CloudFormation template. | DevOps engineer |
| Get variables. | Verify that you have an Amazon ECS cluster with two services running. To retrieve the resource details and store them as variables, use the following commands:
export LoadBalancerDNS=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`LoadBalancerDNS`].OutputValue')
export ECRRepositoryUri=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`ECRRepositoryUri`].OutputValue')
export ECRRepositoryServiceOneUri=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`ECRRepositoryServiceOneUri`].OutputValue')
export ECRRepositoryServiceTwoUri=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`ECRRepositoryServiceTwoUri`].OutputValue')
export ClusterName=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`ClusterName`].OutputValue')
export BucketName=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`BucketName`].OutputValue')
export Service1ListenerArn=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`Service1ListenerArn`].OutputValue')
export Service2ListenerArn=$(aws cloudformation describe-stacks --stack-name ecs-mtls \
--output text \
--query 'Stacks[0].Outputs[?OutputKey==`Service2ListenerArn`].OutputValue')
| DevOps engineer |
| Create a CodeBuild project. | To use a CodeBuild project to create the Docker images for your Amazon ECS services, do the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html)For more details, see [Create a build project in AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/create-project.html) in the AWS documentation. | AWS DevOps, DevOps engineer |
| Build the Docker images. | You can use CodeBuild to perform the image build process. CodeBuild needs permissions to interact with Amazon ECR and to work with Amazon S3.As part of the process, the Docker image is built and pushed to the Amazon ECR registry. For details about the template and the code, see [Additional information](#simplify-application-authentication-with-mutual-tls-in-amazon-ecs-additional).(Optional) To build locally for test purposes, use the following command:# login to ECR
aws ecr get-login-password | docker login --username AWS --password-stdin $ECRRepositoryUri
# build image for service one
cd /service1
aws s3 cp s3://$BucketName/serviceone/ service1/ --recursive
docker build -t $ECRRepositoryServiceOneUri .
docker push $ECRRepositoryServiceOneUri
# build image for service two
cd ../service2
aws s3 cp s3://$BucketName/servicetwo/ service2/ --recursive
docker build -t $ECRRepositoryServiceTwoUri .
docker push $ECRRepositoryServiceTwoUri
| DevOps engineer |
### Enable mutual TLS
| Task | Description | Skills required |
| --- | --- | --- |
| Upload the CA certificate to Amazon S3. | To upload the CA certificate to the Amazon S3 bucket, use the following example command:`aws s3 cp ca-cert.pem s3://$BucketName/acm-trust-store/ ` | AWS DevOps, DevOps engineer |
| Create the trust store. | To create the trust store, use the following example command:TrustStoreArn=`aws elbv2 create-trust-store --name acm-pca-trust-certs \
--ca-certificates-bundle-s3-bucket $BucketName \
--ca-certificates-bundle-s3-key acm-trust-store/ca-cert.pem --query 'TrustStores[].TrustStoreArn' --output text`
| AWS DevOps, DevOps engineer |
| Upload client certificates. | To upload client certificates to Amazon S3 for Docker images, use the following example command:# for service one
aws s3 cp client_cert1.cert s3://$BucketName/serviceone/
aws s3 cp client_private-key1.pem s3://$BucketName/serviceone/
# for service two
aws s3 cp client_cert2.cert s3://$BucketName/servicetwo/
aws s3 cp client_private-key2.pem s3://$BucketName/servicetwo/
| AWS DevOps, DevOps engineer |
| Modify the listener. | To enable mutual TLS on the ALB, modify the HTTPS listeners by using the following commands:aws elbv2 modify-listener \
--listener-arn $Service1ListenerArn \
--certificates CertificateArn=$TLS_CERTIFICATE_ARN_TWO \
--ssl-policy ELBSecurityPolicy-2016-08 \
--protocol HTTPS \
--port 8080 \
--mutual-authentication Mode=verify,TrustStoreArn=$TrustStoreArn,IgnoreClientCertificateExpiry=false
aws elbv2 modify-listener \
--listener-arn $Service2ListenerArn \
--certificates CertificateArn=$TLS_CERTIFICATE_ARN_TWO \
--ssl-policy ELBSecurityPolicy-2016-08 \
--protocol HTTPS \
--port 8090 \
--mutual-authentication Mode=verify,TrustStoreArn=$TrustStoreArn,IgnoreClientCertificateExpiry=false
For more information, see [Configuring mutual TLS on an Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/configuring-mtls-with-elb.html) in the AWS documentation. | AWS DevOps, DevOps engineer |
### Update the services
| Task | Description | Skills required |
| --- | --- | --- |
| Update the Amazon ECS task definition. | To update the Amazon ECS task definition, modify the `image` parameter in the new revision.To get the values for the respective services, update the task definitions with the new Docker images Uri that you built in the previous steps: `echo $ECRRepositoryServiceOneUri` or `echo $ECRRepositoryServiceTwoUri`
"containerDefinitions": [
{
"name": "nginx",
"image": "public.ecr.aws/nginx/nginx:latest", # <----- change to new Uri
"cpu": 0,
For more information, see [Updating an Amazon ECS task definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/update-task-definition-console-v2.html) using the console in the AWS documentation. | AWS DevOps, DevOps engineer |
| Update the Amazon ECS service. | Update the service with the latest task definition. This task definition is the blueprint for the newly built Docker images, and it contains the client certificate that’s required for the mutual TLS authentication. To update the service, use the following procedure:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html)Repeat the steps for the other service. | AWS administrator, AWS DevOps, DevOps engineer |
### Access the application
| Task | Description | Skills required |
| --- | --- | --- |
| Copy the application URL. | Use the Amazon ECS console to view the task. When the task status has been updated to **Running**, select the task. In the **Task **section, copy the task ID. | AWS administrator, AWS DevOps |
| Test your application. | To test your application, use ECS Exec to access the tasks.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/simplify-application-authentication-with-mutual-tls-in-amazon-ecs.html) | AWS administrator, AWS DevOps |
## Related resources
**Amazon ECS documentation**
+ [Creating an Amazon ECS task definition using the console](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html)
+ [Creating a container image for use on Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-container-image.html)
+ [Amazon ECS clusters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html)
+ [Amazon ECS for AWS Fargate](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-container-image.html#create-container-image-next-steps)
+ [Amazon ECS networking best practices](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/networking-best-practices.html)
+ [Amazon ECS service definition parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html)
**Other AWS resources**
+ [How do I use AWS private CA to configure mTLS on the Application Load Balancer?](https://repost.aws/knowledge-center/elb-alb-configure-private-ca-mtls) (AWS re:Post)
## Additional information
**Editing the Dockerfile**** **
The following code shows the commands that you edit in the Dockerfile for service 1:
```
FROM public.ecr.aws/nginx/nginx:latest
WORKDIR /usr/share/nginx/html
RUN echo "Returning response from Service 1: Ok" > /usr/share/nginx/html/index.html
ADD client_cert1.cert client_private-key1.pem /usr/local/share/ca-certificates/
RUN chmod -R 400 /usr/local/share/ca-certificates/
```
The following code shows the commands that you edit in the Dockerfile for service 2:
```
FROM public.ecr.aws/nginx/nginx:latest
WORKDIR /usr/share/nginx/html
RUN echo "Returning response from Service 2: Ok" > /usr/share/nginx/html/index.html
ADD client_cert2.cert client_private-key2.pem /usr/local/share/ca-certificates/
RUN chmod -R 400 /usr/local/share/ca-certificates/
```
If you’re building the Docker images with CodeBuild, the `buildspec` file uses the CodeBuild build number to uniquely identify image versions as a tag value. You can change the `buildspec` file to fit your requirements, as shown in the following `buildspec `custom code:
```
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ECR_REPOSITORY_URI
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- IMAGE_TAG=${COMMIT_HASH:=latest}
build:
commands:
# change the S3 path depending on the service
- aws s3 cp s3://$YOUR_S3_BUCKET_NAME/serviceone/ $CodeBuild_SRC_DIR/ --recursive
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $ECR_REPOSITORY_URI:latest .
- docker tag $ECR_REPOSITORY_URI:latest $ECR_REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $ECR_REPOSITORY_URI:latest
- docker push $ECR_REPOSITORY_URI:$IMAGE_TAG
- echo Writing image definitions file...
# for ECS deployment reference
- printf '[{"name":"%s","imageUri":"%s"}]' $CONTAINER_NAME $ECR_REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
artifacts:
files:
- imagedefinitions.json
```
# More patterns
**Topics**
+ [Automate deletion of AWS CloudFormation stacks and associated resources](automate-deletion-cloudformation-stacks-associated-resources.md)
+ [Automate dynamic pipeline management for deploying hotfix solutions in Gitflow environments by using AWS Service Catalog and AWS CodePipeline](automate-dynamic-pipeline-management-for-deploying-hotfix-solutions.md)
+ [Automatically build CI/CD pipelines and Amazon ECS clusters for microservices using AWS CDK](automatically-build-ci-cd-pipelines-and-amazon-ecs-clusters-for-microservices-using-aws-cdk.md)
+ [Build and push Docker images to Amazon ECR using GitHub Actions and Terraform](build-and-push-docker-images-to-amazon-ecr-using-github-actions-and-terraform.md)
+ [Containerize mainframe workloads that have been modernized by Blu Age](containerize-mainframe-workloads-that-have-been-modernized-by-blu-age.md)
+ [Create a custom log parser for Amazon ECS using a Firelens log router](create-a-custom-log-parser-for-amazon-ecs-using-a-firelens-log-router.md)
+ [Deploy agentic systems on Amazon Bedrock with the CrewAI framework by using Terraform](deploy-agentic-systems-on-amazon-bedrock-with-the-crewai-framework.md)
+ [Deploy an environment for containerized Blu Age applications by using Terraform](deploy-an-environment-for-containerized-blu-age-applications-by-using-terraform.md)
+ [Deploy preprocessing logic into an ML model in a single endpoint using an inference pipeline in Amazon SageMaker](deploy-preprocessing-logic-into-an-ml-model-in-a-single-endpoint-using-an-inference-pipeline-in-amazon-sagemaker.md)
+ [Deploy workloads from Azure DevOps pipelines to private Amazon EKS clusters](deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.md)
+ [Implement AI-powered Kubernetes diagnostics and troubleshooting with K8sGPT and Amazon Bedrock integration](implement-ai-powered-kubernetes-diagnostics-and-troubleshooting-with-k8sgpt-and-amazon-bedrock-integration.md)
+ [Manage blue/green deployments of microservices to multiple accounts and Regions by using AWS code services and AWS KMS multi-Region keys](manage-blue-green-deployments-of-microservices-to-multiple-accounts-and-regions-by-using-aws-code-services-and-aws-kms-multi-region-keys.md)
+ [Manage on-premises container applications by setting up Amazon ECS Anywhere with the AWS CDK](manage-on-premises-container-applications-by-setting-up-amazon-ecs-anywhere-with-the-aws-cdk.md)
+ [Migrate from Oracle WebLogic to Apache Tomcat (TomEE) on Amazon ECS](migrate-from-oracle-weblogic-to-apache-tomcat-tomee-on-amazon-ecs.md)
+ [Modernize ASP.NET Web Forms applications on AWS](modernize-asp-net-web-forms-applications-on-aws.md)
+ [Monitor Amazon ECR repositories for wildcard permissions using AWS CloudFormation and AWS Config](monitor-amazon-ecr-repositories-for-wildcard-permissions-using-aws-cloudformation-and-aws-config.md)
+ [Monitor application activity by using CloudWatch Logs Insights](monitor-application-activity-by-using-cloudwatch-logs-insights.md)
+ [Set up a CI/CD pipeline for hybrid workloads on Amazon ECS Anywhere by using AWS CDK and GitLab](set-up-a-ci-cd-pipeline-for-hybrid-workloads-on-amazon-ecs-anywhere-by-using-aws-cdk-and-gitlab.md)
+ [Set up end-to-end encryption for applications on Amazon EKS using cert-manager and Let's Encrypt](set-up-end-to-end-encryption-for-applications-on-amazon-eks-using-cert-manager-and-let-s-encrypt.md)
+ [Simplify Amazon EKS multi-tenant application deployment by using Flux](simplify-amazon-eks-multi-tenant-application-deployment-by-using-flux.md)
+ [Streamline machine learning workflows from local development to scalable experiments by using SageMaker AI and Hydra](streamline-machine-learning-workflows-by-using-amazon-sagemaker.md)
+ [Structure a Python project in hexagonal architecture using AWS Lambda](structure-a-python-project-in-hexagonal-architecture-using-aws-lambda.md)
+ [Test AWS infrastructure by using LocalStack and Terraform Tests](test-aws-infra-localstack-terraform.md)
+ [Coordinate resource dependency and task execution by using the AWS Fargate WaitCondition hook construct](use-the-aws-fargate-waitcondition-hook-construct.md)
+ [Use Amazon Bedrock agents to automate creation of access entry controls in Amazon EKS through text-based prompts](using-amazon-bedrock-agents-to-automate-creation-of-access-entry-controls-in-amazon-eks.md)