Managing virtual clusters

A virtual cluster is a Kubernetes namespace that Amazon EMR is registered with. You can create, describe, list, and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements. See possible use cases in the Kubernetes Concepts Overview documentation.

To register Amazon EMR with a Kubernetes namespace on an Amazon EKS cluster, you need the name of the EKS cluster and the namespace that has been set up for running your workload. These registered clusters in Amazon EMR are called virtual clusters because they do not manage physical compute or storage but point to a Kubernetes namespace where your workload is scheduled.

Create a virtual cluster

Run the following command to create a virtual cluster by registering Amazon EMR with a namespace on an EKS cluster. Replace virtual_cluster_name with a name that you provide for your virtual cluster. Replace eks_cluster_name with the name of the EKS cluster. Replace the namespace_name with the namespace that you want to register Amazon EMR with.

aws emr-containers create-virtual-cluster \
--name virtual_cluster_name \
--container-provider '{
    "id": "eks_cluster_name",
    "type": "EKS",
    "info": {
        "eksInfo": {
            "namespace": "namespace_name"
        }
    }
}' 

Alternatively, you can create a JSON file that includes the required parameters for the virtual cluster, as the following example demonstrates.

{
    "name": "virtual_cluster_name", 
    "containerProvider": {
        "type": "EKS", 
        "id": "eks_cluster_name", 
        "info": {
            "eksInfo": {
                "namespace": "namespace_name"
            }
        }
    }
}

Then run the following create-virtual-cluster command with the path to the JSON file.

aws emr-containers create-virtual-cluster \
--cli-input-json file://./create-virtual-cluster-request.json
    

List virtual clusters

Run the following command to view the status of virtual clusters.

aws emr-containers list-virtual-clusters

Describe a virtual cluster

Run the following command to get more details about a virtual cluster, such as namespace, status, and date registered. Replace 123456 with your virtual cluster ID.

aws emr-containers describe-virtual-cluster --id 123456

Delete a virtual cluster

Run the following command to delete a virtual cluster. Replace 123456 with your virtual cluster ID.

aws emr-containers delete-virtual-cluster --id 123456

Virtual cluster states

The following table describes the four possible states of a virtual cluster.

Concurrent job limits for virtual clusters

You can configure concurrent job limits on an Amazon EMR on EKS virtual cluster to control how many job runs execute simultaneously and how many can wait in queue. You set the concurrency limit (maxConcurrentJobRuns) and the queue depth (maxInQueueJobRuns) independently, so you can cap running job runs, queued job runs, or both. When you set these limits, the StartJobRun API provides backpressure at the virtual cluster level. Job runs beyond the running limit wait in the queue in the SUBMITTED or PENDING state instead of starting immediately, and once the queue is full, StartJobRun rejects further submissions. For example, if you set a virtual cluster to allow 500 concurrent job runs and 100 queued job runs, the 101st queued submission is rejected, and you can rebalance that workload across other virtual clusters on the same EKS cluster or add capacity. When you have not set a concurrency limit and queue depth keeps growing, so that job runs stay in the SUBMITTED or PENDING state longer before they start, it can signal that the underlying EKS cluster is running low on compute resources and cannot schedule new pods fast enough. In that case, route the workload to another cluster or add capacity.

Concurrent job limits add a control layer in front of the Kubernetes scheduler and the ResourceQuota feature on the Kubernetes website. Because they are enforced at StartJobRun, before any pods are created, excess load is queued or rejected at the API, which protects the underlying cluster before jobs ever reach it. Kubernetes still enforces the actual CPU and memory ceiling underneath.

Key benefits of concurrent job limits

Getting started with concurrent job limits

You configure concurrent job limits with the schedulerConfiguration field on a virtual cluster. This field accepts two parameters:

AWS CLI

To set limits when you create a virtual cluster, specify schedulerConfiguration in your request.

aws emr-containers create-virtual-cluster \
  --name my-virtual-cluster \
  --container-provider '{ ... }' \
  --scheduler-configuration '{
      "maxConcurrentJobRuns": 500,
      "maxInQueueJobRuns": 100
  }'

To change limits on an existing virtual cluster, use the update-virtual-cluster command.

aws emr-containers update-virtual-cluster \
  --id virtual-cluster-id \
  --scheduler-configuration '{
      "maxConcurrentJobRuns": 500,
      "maxInQueueJobRuns": 100
  }'

To remove the limits from a virtual cluster, pass an empty schedulerConfiguration. This clears the configuration, so no limits apply and the virtual cluster returns to default (unlimited) behavior. Note that omitting schedulerConfiguration from the request instead leaves the existing limits unchanged — you must pass an empty object to clear them.

aws emr-containers update-virtual-cluster \
  --id virtual-cluster-id \
  --scheduler-configuration '{}'

To view the current limits and live job counts, use the describe-virtual-cluster command. The response includes both your schedulerConfiguration and a SchedulerStatus object with the current activeJobRunCount and inQueueJobRunCount.

Choosing values for maxConcurrentJobRuns and maxInQueueJobRuns

The right limits depend on three things: how much work your Amazon EKS cluster can run at once, how bursty your submissions are, and how you want the virtual cluster to behave when it is full. Use the following guidance to pick a starting point, and then refine it from the live counters.

Setting maxConcurrentJobRuns (running slots)

maxConcurrentJobRuns is a job-granularity, count-based guardrail. A rough estimate here can protect the underlying Amazon EKS cluster from being degraded due to load and can improve availability.

Setting maxInQueueJobRuns (queue depth)

maxInQueueJobRuns controls how large a backlog the virtual cluster accepts before it starts rejecting submissions. It is a burst-absorption buffer. Consider the following factors.

Considerations for concurrent job limits