Simplify Amazon EKS multi-tenant application deployment by using Flux
Created by Nadeem Rahaman (AWS), Aditya Ambati (AWS), Aniket Dekate (AWS), and Shrikant Patil (AWS)
Summary
Notice: AWS CodeCommit is no longer available to new customers. Existing customers of AWS CodeCommit can continue to use the service as normal. Learn more
Many companies that offer products and services are data-regulated industries that are required to maintain data barriers between their internal business functions. This pattern describes how you can use the multi-tenancy feature in Amazon Elastic Kubernetes Service (Amazon EKS) to build a data platform that achieves logical and physical isolation between tenants or users that share a single Amazon EKS cluster. The pattern provides isolation through the following approaches:
Kubernetes namespace isolation
Role-based access control (RBAC)
Network policies
Resource quotas
AWS Identity and Access Management (IAM) roles for service accounts (IRSA)
In addition, this solution uses Flux to keep the tenant configuration immutable when you deploy applications. You can deploy your tenant applications by specifying the tenant repository that contains the Flux kustomization.yaml
file in your configuration.
This pattern implements the following:
An AWS CodeCommit repository, AWS CodeBuild projects, and an AWS CodePipeline pipeline, which are created by manually deploying Terraform scripts.
Network and compute components required for hosting the tenants. These are created through CodePipeline and CodeBuild by using Terraform.
Tenant namespaces, network policies, and resource quotas, which are configured through a Helm chart.
Applications that belong to different tenants, deployed by using Flux.
We recommend that you carefully plan and build your own architecture for multi-tenancy based on your unique requirements and security considerations. This pattern provides a starting point for your implementation.
Prerequisites and limitations
Prerequisites
An active AWS account
AWS Command Line Interface (AWS CLI) version 2.11.4 or later, installed and configured
Terraform
version 0.12 or later installed on your local machine Terraform AWS Provider
version 3.0.0 or later Kubernetes Provider
version 2.10 or later Helm Provider
version 2.8.0 or later Kubectl Provider
version 1.14 or later
Limitations
Dependency on Terraform manual deployments: The workflow's initial setup, including creating CodeCommit repositories, CodeBuild projects, and CodePipeline pipelines, relies on manual Terraform deployments. This introduces a potential limitation in terms of automation and scalability, because it requires manual intervention for infrastructure changes.
CodeCommit repository dependency: The workflow relies on CodeCommit repositories as the source code management solution and is tightly coupled with AWS services.
Architecture
Target architectures
This pattern deploys three modules to build the pipeline, network, and compute infrastructure for a data platform, as illustrated in the following diagrams.
Pipeline architecture:

Network architecture:

Compute architecture:

Tools
AWS services
AWS CodeBuild is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.
AWS CodeCommit is a version control service that helps you privately store and manage Git repositories, without needing to manage your own source control system.
AWS CodePipeline helps you quickly model and configure the different stages of a software release and automate the steps required to release software changes continuously.
Amazon Elastic Kubernetes Service (Amazon EKS) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.
AWS Transit Gateway is a central hub that connects virtual private clouds (VPCs) and on-premises networks.
Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.
Other tools
Cilium Network Policies
support Kubernetes L3 and L4 networking policies. They can be extended with L7 policies to provide API-level security for HTTP, Kafka, and gRPC, and other similar protocols. Flux
is a Git-based continuous delivery (CD) tool that automates application deployments on Kubernetes. Helm
is an open source package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Terraform
is an infrastructure as code (IaC) tool from HashiCorp that helps you create and manage cloud and on-premises resources.
Code repository
The code for this pattern is available in the GitHub EKS Multi-Tenancy Terraform Solution
Best practices
For guidelines and best practices for using this implementation, see the following:
Epics
Task | Description | Skills required |
---|---|---|
Clone the project repository. | Clone the GitHub EKS Multi-Tenancy Terraform Solution
| AWS DevOps |
Bootstrap the Terraform S3 bucket and Amazon DynamoDB. |
| AWS DevOps |
Update the |
| AWS DevOps |
Deploy the pipeline module. | To create pipeline resources, run the following Terraform commands manually. There is no orchestration for running these commands automatically.
| AWS DevOps |
Task | Description | Skills required |
---|---|---|
Start the pipeline. |
After this first run, the pipeline starts automatically whenever you commit a change to the CodeCommit repository main branch. The pipeline includes the following stages:
| AWS DevOps |
Validate the resources created through the network module. | Confirm that the following AWS resources were created after the pipeline deployed successfully:
| AWS DevOps |
Task | Description | Skills required |
---|---|---|
Update | To deploy the add-ons for the Amazon EKS private cluster, the CodeBuild project must be attached to the Amazon EKS VPC.
| AWS DevOps |
Update the | In the
| AWS DevOps |
Update the |
| AWS DevOps |
Validate compute resources. | After you update the files in the previous steps, CodePipeline starts automatically. Confirm that it created the following AWS resources for the compute infrastructure:
| AWS DevOps |
Task | Description | Skills required |
---|---|---|
Validate the tenant management resources in Kubernetes. | Run the following commands to check that tenant management resources were created successfully with the help of Helm.
| AWS DevOps |
Verify tenant application deployments. | Run the following commands to verify that the tenant applications were deployed.
|
Troubleshooting
Issue | Solution |
---|---|
You encounter an error message that’s similar to the following:
| Follow these steps to troubleshoot the issue:
|
Related resources
Additional information
Here's an example repository structure for deploying tenant applications:
applications
sample_tenant_app
├── README.md
├── base
│ ├── configmap.yaml
│ ├── deployment.yaml
│ ├── ingress.yaml
│ ├── kustomization.yaml
│ └── service.yaml
└── overlays
├── tenant-1
│ ├── configmap.yaml
│ ├── deployment.yaml
│ └── kustomization.yaml
└── tenant-2
├── configmap.yaml
└── kustomization.yaml