WorkerConfiguration

class aws_cdk.aws_glue_alpha.WorkerConfiguration(*, number_of_workers, worker_type)

Bases: object

(experimental) The worker configuration for a Spark job.

The worker type and the number of workers are set together: providing this configuration requires both values, so a Spark job can never be given one without the other.

Parameters:

• number_of_workers (Union[int, float]) – (experimental) The number of workers of the given workerType that are allocated when a job runs.

• worker_type (WorkerType) – (experimental) The type of predefined worker that is allocated when a job runs. Enum options: Standard, G_1X, G_2X, G_025X, G_4X, G_8X, Z_2X

Stability:

experimental

ExampleMetadata:

infused

Example:

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkEtlJob(stack, "PySparkETLJob",
    job_name="PySparkETLJobCustomName",
    description="This is a description",
    role=role,
    script=script,
    glue_version=glue.GlueVersion.V5_1,
    continuous_logging=glue.ContinuousLoggingProps(enabled=False),
    worker_configuration=glue.WorkerConfiguration(
        worker_type=glue.WorkerType.G_2X,
        number_of_workers=2
    ),
    max_concurrent_runs=100,
    timeout=cdk.Duration.hours(2),
    connections=[glue.Connection.from_connection_name(stack, "Connection", "connectionName")],
    security_configuration=glue.SecurityConfiguration.from_security_configuration_name(stack, "SecurityConfig", "securityConfigName"),
    tags={
        "FirstTagName": "FirstTagValue",
        "SecondTagName": "SecondTagValue",
        "XTagName": "XTagValue"
    },
    max_retries=2
)

Attributes

number_of_workers

(experimental) The number of workers of the given workerType that are allocated when a job runs.

Stability:

experimental

worker_type

(experimental) The type of predefined worker that is allocated when a job runs.

Enum options: Standard, G_1X, G_2X, G_025X, G_4X, G_8X, Z_2X

Stability:

experimental