Managing virtual clusters

A virtual cluster is a Kubernetes namespace that Amazon EMR is registered with. You can create, describe, list, and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements. See possible use cases in the Kubernetes Concepts Overview documentation.

To register Amazon EMR with a Kubernetes namespace on an Amazon EKS cluster, you need the name of the EKS cluster and the namespace that has been set up for running your workload. These registered clusters in Amazon EMR are called virtual clusters because they do not manage physical compute or storage but point to a Kubernetes namespace where your workload is scheduled.

Create a virtual cluster

Run the following command to create a virtual cluster by registering Amazon EMR with a namespace on an EKS cluster. Replace virtual_cluster_name with a name that you provide for your virtual cluster. Replace eks_cluster_name with the name of the EKS cluster. Replace the namespace_name with the namespace that you want to register Amazon EMR with.

aws emr-containers create-virtual-cluster \
--name virtual_cluster_name \
--container-provider '{
    "id": "eks_cluster_name",
    "type": "EKS",
    "info": {
        "eksInfo": {
            "namespace": "namespace_name"
        }
    }
}' 

Alternatively, you can create a JSON file that includes the required parameters for the virtual cluster, as the following example demonstrates.

{
    "name": "virtual_cluster_name", 
    "containerProvider": {
        "type": "EKS", 
        "id": "eks_cluster_name", 
        "info": {
            "eksInfo": {
                "namespace": "namespace_name"
            }
        }
    }
}

Then run the following create-virtual-cluster command with the path to the JSON file.

aws emr-containers create-virtual-cluster \
--cli-input-json file://./create-virtual-cluster-request.json
    

List virtual clusters

Run the following command to view the status of virtual clusters.

aws emr-containers list-virtual-clusters

Describe a virtual cluster

Run the following command to get more details about a virtual cluster, such as namespace, status, and date registered. Replace 123456 with your virtual cluster ID.

aws emr-containers describe-virtual-cluster --id 123456

Delete a virtual cluster

Run the following command to delete a virtual cluster. Replace 123456 with your virtual cluster ID.

aws emr-containers delete-virtual-cluster --id 123456

Virtual cluster states

The following table describes the four possible states of a virtual cluster.