Pilih preferensi cookie Anda

Kami menggunakan cookie penting serta alat serupa yang diperlukan untuk menyediakan situs dan layanan. Kami menggunakan cookie performa untuk mengumpulkan statistik anonim sehingga kami dapat memahami cara pelanggan menggunakan situs dan melakukan perbaikan. Cookie penting tidak dapat dinonaktifkan, tetapi Anda dapat mengklik “Kustom” atau “Tolak” untuk menolak cookie performa.

Jika Anda setuju, AWS dan pihak ketiga yang disetujui juga akan menggunakan cookie untuk menyediakan fitur situs yang berguna, mengingat preferensi Anda, dan menampilkan konten yang relevan, termasuk iklan yang relevan. Untuk menerima atau menolak semua cookie yang tidak penting, klik “Terima” atau “Tolak”. Untuk membuat pilihan yang lebih detail, klik “Kustomisasi”.

Simplify Amazon EKS multi-tenant application deployment by using Flux - AWS Prescriptive Guidance
Halaman ini belum diterjemahkan ke dalam bahasa Anda. Minta terjemahan

Simplify Amazon EKS multi-tenant application deployment by using Flux

Created by Nadeem Rahaman (AWS), Aditya Ambati (AWS), Aniket Dekate (AWS), and Shrikant Patil (AWS)

Summary

Notice: AWS CodeCommit is no longer available to new customers. Existing customers of AWS CodeCommit can continue to use the service as normal. Learn more

Many companies that offer products and services are data-regulated industries that are required to maintain data barriers between their internal business functions. This pattern describes how you can use the multi-tenancy feature in Amazon Elastic Kubernetes Service (Amazon EKS) to build a data platform that achieves logical and physical isolation between tenants or users that share a single Amazon EKS cluster. The pattern provides isolation through the following approaches:

  • Kubernetes namespace isolation

  • Role-based access control (RBAC)

  • Network policies

  • Resource quotas

  • AWS Identity and Access Management (IAM) roles for service accounts (IRSA)

In addition, this solution uses Flux to keep the tenant configuration immutable when you deploy applications. You can deploy your tenant applications by specifying the tenant repository that contains the Flux kustomization.yaml file in your configuration.

This pattern implements the following:

  • An AWS CodeCommit repository, AWS CodeBuild projects, and an AWS CodePipeline pipeline, which are created by manually deploying Terraform scripts.

  • Network and compute components required for hosting the tenants. These are created through CodePipeline and CodeBuild by using Terraform.

  • Tenant namespaces, network policies, and resource quotas, which are configured through a Helm chart.

  • Applications that belong to different tenants, deployed by using Flux.

We recommend that you carefully plan and build your own architecture for multi-tenancy based on your unique requirements and security considerations. This pattern provides a starting point for your implementation.

Prerequisites and limitations

Prerequisites

Limitations

  • Dependency on Terraform manual deployments: The workflow's initial setup, including creating CodeCommit repositories, CodeBuild projects, and CodePipeline pipelines, relies on manual Terraform deployments. This introduces a potential limitation in terms of automation and scalability, because it requires manual intervention for infrastructure changes.

  • CodeCommit repository dependency: The workflow relies on CodeCommit repositories as the source code management solution and is tightly coupled with AWS services.

Architecture

Target architectures

This pattern deploys three modules to build the pipeline, network, and compute infrastructure for a data platform, as illustrated in the following diagrams.

Pipeline architecture:

Pipeline infrastructure for Amazon EKS multi-tenant architecture

Network architecture:

Network infrastructure for Amazon EKS multi-tenant architecture

Compute architecture:

Compute infrastructure for Amazon EKS multi-tenant architecture

Tools

AWS services

  • AWS CodeBuild is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.

  • AWS CodeCommit is a version control service that helps you privately store and manage Git repositories, without needing to manage your own source control system.

  • AWS CodePipeline helps you quickly model and configure the different stages of a software release and automate the steps required to release software changes continuously.

  • Amazon Elastic Kubernetes Service (Amazon EKS) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.

  • AWS Transit Gateway is a central hub that connects virtual private clouds (VPCs) and on-premises networks.

  • Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

Other tools

  • Cilium Network Policies support Kubernetes L3 and L4 networking policies. They can be extended with L7 policies to provide API-level security for HTTP, Kafka, and gRPC, and other similar protocols.

  • Flux is a Git-based continuous delivery (CD) tool that automates application deployments on Kubernetes.

  • Helm is an open source package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.

  • Terraform is an infrastructure as code (IaC) tool from HashiCorp that helps you create and manage cloud and on-premises resources.

Code repository

The code for this pattern is available in the GitHub EKS Multi-Tenancy Terraform Solution repository.

Best practices

For guidelines and best practices for using this implementation, see the following:

Epics

TaskDescriptionSkills required

Clone the project repository.

Clone the GitHub EKS Multi-Tenancy Terraform Solution repository by running the following command in a terminal window:

git clone https://github.com/aws-samples/aws-eks-multitenancy-deployment.git
AWS DevOps

Bootstrap the Terraform S3 bucket and Amazon DynamoDB.

  1. In the bootstrap folder, open the bootstrap.sh file and update the variable values for the S3 bucket name, DynamoDB table name, and AWS Region:

    S3_BUCKET_NAME="<S3_BUCKET_NAME>" DYNAMODB_TABLE_NAME="<DYNAMODB_NAME>" REGION="<AWS_REGION>"
  2. Run the bootstrap.sh script. The script requires the AWS CLI, which you installed as part of prerequisites.

    cd bootstrap ./bootstrap.sh
AWS DevOps

Update the run.sh and locals.tf files.

  1. After the bootstrap process completes successfully, copy the S3 bucket and DynamoDB table name from the variables section of the bootstrap.sh script:

    # Variables S3_BUCKET_NAME="<S3_BUCKET_NAME>" DYNAMODB_TABLE_NAME="<DYNAMODB_NAME"
  2. Paste those values to the run.sh script, which is in the root directory of the project:

    BACKEND_BUCKET_ID="<SAME_NAME_AS_S3_BUCKET_NAME>" DYNAMODB_ID="<SAME_NAME_AS_DYNAMODB_NAME>"
  3. Upload the project code to a CodeCommit repository. You can automatically create this repository through Terraform by setting the following variable to true in the demo/pipeline/locals.tf file:

    create_new_repo = true
  4. Update the locals.tf file according to your requirements to create pipeline resources.

AWS DevOps

Deploy the pipeline module.

To create pipeline resources, run the following Terraform commands manually. There is no orchestration for running these commands automatically.

./run.sh -m pipeline -e demo -r <AWS_REGION> -t init ./run.sh -m pipeline -e demo -r <AWS_REGION> -t plan ./run.sh -m pipeline -e demo -r <AWS_REGION> -t apply
AWS DevOps

Create pipelines for Terraform build, test, and deploy stages

TaskDescriptionSkills required

Clone the project repository.

Clone the GitHub EKS Multi-Tenancy Terraform Solution repository by running the following command in a terminal window:

git clone https://github.com/aws-samples/aws-eks-multitenancy-deployment.git
AWS DevOps

Bootstrap the Terraform S3 bucket and Amazon DynamoDB.

  1. In the bootstrap folder, open the bootstrap.sh file and update the variable values for the S3 bucket name, DynamoDB table name, and AWS Region:

    S3_BUCKET_NAME="<S3_BUCKET_NAME>" DYNAMODB_TABLE_NAME="<DYNAMODB_NAME>" REGION="<AWS_REGION>"
  2. Run the bootstrap.sh script. The script requires the AWS CLI, which you installed as part of prerequisites.

    cd bootstrap ./bootstrap.sh
AWS DevOps

Update the run.sh and locals.tf files.

  1. After the bootstrap process completes successfully, copy the S3 bucket and DynamoDB table name from the variables section of the bootstrap.sh script:

    # Variables S3_BUCKET_NAME="<S3_BUCKET_NAME>" DYNAMODB_TABLE_NAME="<DYNAMODB_NAME"
  2. Paste those values to the run.sh script, which is in the root directory of the project:

    BACKEND_BUCKET_ID="<SAME_NAME_AS_S3_BUCKET_NAME>" DYNAMODB_ID="<SAME_NAME_AS_DYNAMODB_NAME>"
  3. Upload the project code to a CodeCommit repository. You can automatically create this repository through Terraform by setting the following variable to true in the demo/pipeline/locals.tf file:

    create_new_repo = true
  4. Update the locals.tf file according to your requirements to create pipeline resources.

AWS DevOps

Deploy the pipeline module.

To create pipeline resources, run the following Terraform commands manually. There is no orchestration for running these commands automatically.

./run.sh -m pipeline -e demo -r <AWS_REGION> -t init ./run.sh -m pipeline -e demo -r <AWS_REGION> -t plan ./run.sh -m pipeline -e demo -r <AWS_REGION> -t apply
AWS DevOps
TaskDescriptionSkills required

Start the pipeline.

  1. In the templates folder, make sure that the buildspec files have the following variable set to network:

    TF_MODULE_TO_BUILD: "network"
  2. On the CodePipeline console, on the pipeline details page, start the pipeline by choosing Release change.

After this first run, the pipeline starts automatically whenever you commit a change to the CodeCommit repository main branch.

The pipeline includes the following stages:

  • validate initializes Terraform, runs Terraform security scans by using the checkov and tfsec tools, and uploads the scan reports to the S3 bucket.

  • plan shows the Terraform plan and uploads the plan to the S3 bucket.

  • apply applies the Terraform plan output from the S3 bucket and creates AWS resources.

  • destroy removes the AWS resources created during the apply stage. To enable this optional stage, set the following variable to true in the demo/pipeline/locals.tf file:

    enable_destroy_stage = true
AWS DevOps

Validate the resources created through the network module.

Confirm that the following AWS resources were created after the pipeline deployed successfully:

  • An egress VPC with three public and three private subnets, internet gateway, and NAT gateway.

  • An Amazon EKS VPC with three private subnets.

  • Tenant 1 and Tenant 2 VPCs with three private subnets each.

  • A transit gateway with all VPC attachments and routes to each private subnet.

  • A static transit gateway route for the Amazon EKS egress VPC with a destination CIDR block of 0.0.0.0/0. This is required to enable all VPCs to have outbound internet access through the Amazon EKS egress VPC.

AWS DevOps

Create the network infrastructure

TaskDescriptionSkills required

Start the pipeline.

  1. In the templates folder, make sure that the buildspec files have the following variable set to network:

    TF_MODULE_TO_BUILD: "network"
  2. On the CodePipeline console, on the pipeline details page, start the pipeline by choosing Release change.

After this first run, the pipeline starts automatically whenever you commit a change to the CodeCommit repository main branch.

The pipeline includes the following stages:

  • validate initializes Terraform, runs Terraform security scans by using the checkov and tfsec tools, and uploads the scan reports to the S3 bucket.

  • plan shows the Terraform plan and uploads the plan to the S3 bucket.

  • apply applies the Terraform plan output from the S3 bucket and creates AWS resources.

  • destroy removes the AWS resources created during the apply stage. To enable this optional stage, set the following variable to true in the demo/pipeline/locals.tf file:

    enable_destroy_stage = true
AWS DevOps

Validate the resources created through the network module.

Confirm that the following AWS resources were created after the pipeline deployed successfully:

  • An egress VPC with three public and three private subnets, internet gateway, and NAT gateway.

  • An Amazon EKS VPC with three private subnets.

  • Tenant 1 and Tenant 2 VPCs with three private subnets each.

  • A transit gateway with all VPC attachments and routes to each private subnet.

  • A static transit gateway route for the Amazon EKS egress VPC with a destination CIDR block of 0.0.0.0/0. This is required to enable all VPCs to have outbound internet access through the Amazon EKS egress VPC.

AWS DevOps
TaskDescriptionSkills required

Update locals.tf to enable the CodeBuild project’s access to the VPC.

To deploy the add-ons for the Amazon EKS private cluster, the CodeBuild project must be attached to the Amazon EKS VPC.

  1. In the demo/pipeline folder, open the locals.tf file, and set the vpc_enabled variable to true.

  2. Run the run.sh script to apply the changes to the pipeline module:

    demo/pipeline/locals.tf ./run.sh -m pipeline -env demo -region <AWS_REGION> -tfcmd init ./run.sh -m pipeline -env demo -region <AWS_REGION> -tfcmd plan ./run.sh -m pipeline -env demo -region <AWS_REGION> -tfcmd apply
AWS DevOps

Update the buildspec files to build the compute module.

In the templates folder, in all buildspec YAML files, set the value of the TF_MODULE_TO_BUILD variable from network to compute:

TF_MODULE_TO_BUILD: "compute"
AWS DevOps

Update the values file for the tenant management Helm chart.

  1. Open the values.yaml file in the following location:

    cd cfg-terraform/demo/compute/cfg-tenant-mgmt

    The file looks like this:

    --- global: clusterRoles: operator: platform-tenant flux: flux-tenant-applier flux: tenantCloneBaseUrl: ${TEANT_BASE_URL} repoSecret: ${TENANT_REPO_SECRET} tenants: tenant-1: quotas: limits: cpu: 1 memory: 1Gi flux: path: overlays/tenant-1 tenant-2: quotas: limits: cpu: 1 memory: 2Gi flux: path: overlays/tenant-2
  2. In the global and tenants sections, update the configuration based on your requirements:

    • tenantCloneBaseUrl – Path to the repository that hosts the code for all tenants (we use the same Git repository for all tenants)

    • repoSecret – Kubernetes secret that holds the SSH keys and known hosts to authenticate to the global tenant Git repository

    • quotas – Kubernetes resources quotas that you want to apply for each tenant

    • flux path – Path to the tenant application YAML files in the global tenant repository

AWS DevOps

Validate compute resources.

After you update the files in the previous steps, CodePipeline starts automatically. Confirm that it created the following AWS resources for the compute infrastructure:

  • Amazon EKS cluster with private endpoint

  • Amazon EKS worker nodes

  • Amazon EKS add-ons: external secrets, aws-loadbalancer-controller, and metrics-server

  • GitOps module, Flux Helm chart, Cilium Helm chart, and tenant management Helm chart

AWS DevOps

Create the compute infrastructure

TaskDescriptionSkills required

Update locals.tf to enable the CodeBuild project’s access to the VPC.

To deploy the add-ons for the Amazon EKS private cluster, the CodeBuild project must be attached to the Amazon EKS VPC.

  1. In the demo/pipeline folder, open the locals.tf file, and set the vpc_enabled variable to true.

  2. Run the run.sh script to apply the changes to the pipeline module:

    demo/pipeline/locals.tf ./run.sh -m pipeline -env demo -region <AWS_REGION> -tfcmd init ./run.sh -m pipeline -env demo -region <AWS_REGION> -tfcmd plan ./run.sh -m pipeline -env demo -region <AWS_REGION> -tfcmd apply
AWS DevOps

Update the buildspec files to build the compute module.

In the templates folder, in all buildspec YAML files, set the value of the TF_MODULE_TO_BUILD variable from network to compute:

TF_MODULE_TO_BUILD: "compute"
AWS DevOps

Update the values file for the tenant management Helm chart.

  1. Open the values.yaml file in the following location:

    cd cfg-terraform/demo/compute/cfg-tenant-mgmt

    The file looks like this:

    --- global: clusterRoles: operator: platform-tenant flux: flux-tenant-applier flux: tenantCloneBaseUrl: ${TEANT_BASE_URL} repoSecret: ${TENANT_REPO_SECRET} tenants: tenant-1: quotas: limits: cpu: 1 memory: 1Gi flux: path: overlays/tenant-1 tenant-2: quotas: limits: cpu: 1 memory: 2Gi flux: path: overlays/tenant-2
  2. In the global and tenants sections, update the configuration based on your requirements:

    • tenantCloneBaseUrl – Path to the repository that hosts the code for all tenants (we use the same Git repository for all tenants)

    • repoSecret – Kubernetes secret that holds the SSH keys and known hosts to authenticate to the global tenant Git repository

    • quotas – Kubernetes resources quotas that you want to apply for each tenant

    • flux path – Path to the tenant application YAML files in the global tenant repository

AWS DevOps

Validate compute resources.

After you update the files in the previous steps, CodePipeline starts automatically. Confirm that it created the following AWS resources for the compute infrastructure:

  • Amazon EKS cluster with private endpoint

  • Amazon EKS worker nodes

  • Amazon EKS add-ons: external secrets, aws-loadbalancer-controller, and metrics-server

  • GitOps module, Flux Helm chart, Cilium Helm chart, and tenant management Helm chart

AWS DevOps
TaskDescriptionSkills required

Validate the tenant management resources in Kubernetes.

Run the following commands to check that tenant management resources were created successfully with the help of Helm.

  1. Tenant namespaces were created, as specified in values.yaml:

    kubectl get ns -A
  2. Quotas are assigned to each tenant namespace, as specified in values.yaml:

    kubectl get quota --namespace=<tenant_namespace>
  3. Details of quotas are correct for each tenant namespace:

    kubectl describe quota cpu-memory-resource-quota-limit -n <tenant_namespace>
  4. Cilium Network Policies were applied to each tenant namespace:

    kubectl get CiliumNetworkPolicy -A
AWS DevOps

Verify tenant application deployments.

Run the following commands to verify that the tenant applications were deployed.

  1. Flux is able to connect to the CodeCommit repository that's specified in the GitOps module:

    kubectl get gitrepositories -A
  2. The Flux kustomization controller has deployed the YAML files in the CodeCommit repository:

    kubectl get kustomizations -A
  3. All application resources are deployed in their tenant namespaces:

    kubectl get all -n <tenant_namespace>
  4. An ingress has been created for each tenant:

    kubectl get ingress -n <tenant_namespace>

Check tenant management and other resources

TaskDescriptionSkills required

Validate the tenant management resources in Kubernetes.

Run the following commands to check that tenant management resources were created successfully with the help of Helm.

  1. Tenant namespaces were created, as specified in values.yaml:

    kubectl get ns -A
  2. Quotas are assigned to each tenant namespace, as specified in values.yaml:

    kubectl get quota --namespace=<tenant_namespace>
  3. Details of quotas are correct for each tenant namespace:

    kubectl describe quota cpu-memory-resource-quota-limit -n <tenant_namespace>
  4. Cilium Network Policies were applied to each tenant namespace:

    kubectl get CiliumNetworkPolicy -A
AWS DevOps

Verify tenant application deployments.

Run the following commands to verify that the tenant applications were deployed.

  1. Flux is able to connect to the CodeCommit repository that's specified in the GitOps module:

    kubectl get gitrepositories -A
  2. The Flux kustomization controller has deployed the YAML files in the CodeCommit repository:

    kubectl get kustomizations -A
  3. All application resources are deployed in their tenant namespaces:

    kubectl get all -n <tenant_namespace>
  4. An ingress has been created for each tenant:

    kubectl get ingress -n <tenant_namespace>

Troubleshooting

IssueSolution

You encounter an error message that’s similar to the following:

Failed to checkout and determine revision: unable to clone unknown error: You have successfully authenticated over SSH. You can use Git to interact with AWS CodeCommit.

Follow these steps to troubleshoot the issue:

  1. Verify the tenant application repository: An empty or misconfigured repository might be causing the error. Make sure that the tenant application repository contains the required code.

  2. Redeploy the tenant_mgmt module: In the tenant_mgmt module configuration file, locate the app block, and then set the deploy parameter to 0:

    deploy = 0

    After you run the Terraform apply command, change the deploy parameter value back to 1:

    deploy = 1
  3. Recheck the status: After you run the previous steps, use the following command to check whether the issue persists:

     kubectl get gitrepositories -A

    If it persists, consider diving deeper into the Flux logs for more details or refer to the Flux general troubleshooting guide.

Related resources

Additional information

Here's an example repository structure for deploying tenant applications:

applications sample_tenant_app ├── README.md ├── base │ ├── configmap.yaml │ ├── deployment.yaml │ ├── ingress.yaml │ ├── kustomization.yaml │ └── service.yaml └── overlays ├── tenant-1 │ ├── configmap.yaml │ ├── deployment.yaml │ └── kustomization.yaml └── tenant-2 ├── configmap.yaml └── kustomization.yaml
PrivasiSyarat situsPreferensi cookie
© 2025, Amazon Web Services, Inc. atau afiliasinya. Semua hak dilindungi undang-undang.