Class ScalaSparkFlexEtlJobProps

Properties for a ScalaSparkFlexEtlJob.

Inheritance

object

ScalaSparkFlexEtlJobProps

Implements

IScalaSparkFlexEtlJobProps

ISparkJobProps

IJobProps

Inherited Members

object.GetType()

object.MemberwiseClone()

object.ToString()

object.Equals(object)

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetHashCode()

Namespace: Amazon.CDK.AWS.Glue

Assembly: Amazon.CDK.Lib.dll

Syntax (csharp)

public class ScalaSparkFlexEtlJobProps : IScalaSparkFlexEtlJobProps, ISparkJobProps, IJobProps

Syntax (vb)

Public Class ScalaSparkFlexEtlJobProps Implements IScalaSparkFlexEtlJobProps, ISparkJobProps, IJobProps

Remarks

ExampleMetadata: fixture=_generated

Examples

// The code below shows an example of how to instantiate this type.
            // The values are placeholders you should change.
            using Amazon.CDK;
            using Amazon.CDK.AWS.Glue;
            using Amazon.CDK.AWS.IAM;
            using Amazon.CDK.AWS.Logs;
            using Amazon.CDK.AWS.S3;

            Bucket bucket;
            Code code;
            Connection connection;
            LogGroup logGroup;
            Role role;
            SecurityConfiguration securityConfiguration;

            var scalaSparkFlexEtlJobProps = new ScalaSparkFlexEtlJobProps {
                ClassName = "className",
                Role = role,
                Script = code,

                // the properties below are optional
                Connections = new [] { connection },
                ContinuousLogging = new ContinuousLoggingProps {
                    Enabled = false,

                    // the properties below are optional
                    ConversionPattern = "conversionPattern",
                    LogGroup = logGroup,
                    LogStreamPrefix = "logStreamPrefix",
                    Quiet = false
                },
                DefaultArguments = new Dictionary<string, string> {
                    { "defaultArgumentsKey", "defaultArguments" }
                },
                Description = "description",
                EnableMetrics = false,
                EnableObservabilityMetrics = false,
                ExtraFiles = new [] { code },
                ExtraJars = new [] { code },
                ExtraJarsFirst = false,
                GlueVersion = GlueVersion.V0_9,
                JobName = "jobName",
                MaxConcurrentRuns = 123,
                MaxRetries = 123,
                NotifyDelayAfter = Duration.Minutes(30),
                SecurityConfiguration = securityConfiguration,
                SparkUI = new SparkUIProps {
                    Bucket = bucket,
                    Prefix = "prefix"
                },
                Tags = new Dictionary<string, string> {
                    { "tagsKey", "tags" }
                },
                Timeout = Duration.Minutes(30),
                WorkerConfiguration = new WorkerConfiguration {
                    NumberOfWorkers = 123,
                    WorkerType = WorkerType.STANDARD
                }
            };

Synopsis

Constructors

ScalaSparkFlexEtlJobProps()

Properties for a ScalaSparkFlexEtlJob.

Properties

ClassName	The fully qualified Scala class name that serves as the entry point for the job.
Connections	Connections (optional) List of connections to use for this Glue job Connections are used to connect to other AWS Service or resources within a VPC.
ContinuousLogging	Enables continuous logging with the specified props.
DefaultArguments	Default Arguments (optional) The default arguments for every run of this Glue job, specified as name-value pairs.
Description	Description (optional) Developer-specified description of the Glue job.
EnableMetrics	Enable profiling metrics for the Glue job.
EnableObservabilityMetrics	Enable observability metrics for the Glue job.
ExtraFiles	Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.
ExtraJars	Extra Jars S3 URL (optional) S3 URL where additional jar dependencies are located.
ExtraJarsFirst	Setting this value to true prioritizes the customer's extra JAR files in the classpath.
GlueVersion	Glue Version The version of Glue to use to execute this job.
JobName	Name of the Glue job (optional) Developer-specified name of the Glue job.
MaxConcurrentRuns	Max Concurrent Runs (optional) The maximum number of runs this Glue job can concurrently run.
MaxRetries	Max Retries (optional) Maximum number of retry attempts Glue performs if the job fails.
NotifyDelayAfter	Specifies configuration properties of a notification (optional).
Role	IAM Role (required) IAM Role to use for Glue job execution Must be specified by the developer because the L2 doesn't have visibility into the actions the script(s) takes during the job execution The role must trust the Glue service principal (glue.amazonaws.com) and be granted sufficient permissions.
Script	Script Code Location (required) Script to run when the Glue job executes.
SecurityConfiguration	Security Configuration (optional) Defines the encryption options for the Glue job.
SparkUI	Enables the Spark UI debugging and monitoring with the specified props.
Tags	Tags (optional) A list of key:value pairs of tags to apply to this Glue job resources.
Timeout	Timeout (optional) The maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status.
WorkerConfiguration	The worker type and the number of workers allocated when a job runs.

Constructors

ScalaSparkFlexEtlJobProps()

Properties for a ScalaSparkFlexEtlJob.

public ScalaSparkFlexEtlJobProps()

Remarks

ExampleMetadata: fixture=_generated

Examples

// The code below shows an example of how to instantiate this type.
            // The values are placeholders you should change.
            using Amazon.CDK;
            using Amazon.CDK.AWS.Glue;
            using Amazon.CDK.AWS.IAM;
            using Amazon.CDK.AWS.Logs;
            using Amazon.CDK.AWS.S3;

            Bucket bucket;
            Code code;
            Connection connection;
            LogGroup logGroup;
            Role role;
            SecurityConfiguration securityConfiguration;

            var scalaSparkFlexEtlJobProps = new ScalaSparkFlexEtlJobProps {
                ClassName = "className",
                Role = role,
                Script = code,

                // the properties below are optional
                Connections = new [] { connection },
                ContinuousLogging = new ContinuousLoggingProps {
                    Enabled = false,

                    // the properties below are optional
                    ConversionPattern = "conversionPattern",
                    LogGroup = logGroup,
                    LogStreamPrefix = "logStreamPrefix",
                    Quiet = false
                },
                DefaultArguments = new Dictionary<string, string> {
                    { "defaultArgumentsKey", "defaultArguments" }
                },
                Description = "description",
                EnableMetrics = false,
                EnableObservabilityMetrics = false,
                ExtraFiles = new [] { code },
                ExtraJars = new [] { code },
                ExtraJarsFirst = false,
                GlueVersion = GlueVersion.V0_9,
                JobName = "jobName",
                MaxConcurrentRuns = 123,
                MaxRetries = 123,
                NotifyDelayAfter = Duration.Minutes(30),
                SecurityConfiguration = securityConfiguration,
                SparkUI = new SparkUIProps {
                    Bucket = bucket,
                    Prefix = "prefix"
                },
                Tags = new Dictionary<string, string> {
                    { "tagsKey", "tags" }
                },
                Timeout = Duration.Minutes(30),
                WorkerConfiguration = new WorkerConfiguration {
                    NumberOfWorkers = 123,
                    WorkerType = WorkerType.STANDARD
                }
            };

Properties

ClassName

The fully qualified Scala class name that serves as the entry point for the job.

public string ClassName { get; set; }

Property Value

string

Remarks

See: --class in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Connections

Connections (optional) List of connections to use for this Glue job Connections are used to connect to other AWS Service or resources within a VPC.

public IConnection[]? Connections { get; set; }

Property Value

IConnection[]

Remarks

Default: [] - no connections are added to the job

ContinuousLogging

Enables continuous logging with the specified props.

public IContinuousLoggingProps? ContinuousLogging { get; set; }

Property Value

IContinuousLoggingProps

Remarks

Default: - continuous logging is enabled.

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

DefaultArguments

Default Arguments (optional) The default arguments for every run of this Glue job, specified as name-value pairs.

public IDictionary<string, string>? DefaultArguments { get; set; }

Property Value

IDictionary<string, string>

Remarks

This map is the escape hatch for Glue job arguments that this construct does not model. It MUST NOT be used to set arguments that already have a dedicated prop — configure those through the corresponding prop instead (continuousLogging, enableMetrics, enableObservabilityMetrics, sparkUI, className, extraJars, extraJarsFirst, extraPythonFiles, extraFiles). Passing a construct-managed argument (e.g. --enable-continuous-cloudwatch-log, --enable-metrics, --enable-spark-ui, --job-language) or a Glue-reserved argument (--debug, --mode, --JOB_NAME, --endpoint) here throws at synthesis time, so there is exactly one way to express each intent.

Also note that these are emitted verbatim into the CloudFormation template, so avoid placing secrets here in plaintext. Pass secrets to the job at runtime through AWS Secrets Manager instead. A synthesis-time warning is emitted when an argument key looks like a credential and holds a plaintext literal.

Default: - no arguments

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html for a list of reserved parameters

Description

Description (optional) Developer-specified description of the Glue job.

public string? Description { get; set; }

Property Value

string

Remarks

Default: - no value

EnableMetrics

Enable profiling metrics for the Glue job.

public bool? EnableMetrics { get; set; }

Property Value

bool?

Remarks

When enabled, adds '--enable-metrics' to job arguments.

Default: true

EnableObservabilityMetrics

Enable observability metrics for the Glue job.

public bool? EnableObservabilityMetrics { get; set; }

Property Value

bool?

Remarks

When enabled, adds '--enable-observability-metrics': 'true' to job arguments.

Default: true

ExtraFiles

Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.

public Code[]? ExtraFiles { get; set; }

Property Value

Code[]

Remarks

Default: - no extra files specified.

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

ExtraJars

Extra Jars S3 URL (optional) S3 URL where additional jar dependencies are located.

public Code[]? ExtraJars { get; set; }

Property Value

Code[]

Remarks

Default: - no extra jar files

ExtraJarsFirst

Setting this value to true prioritizes the customer's extra JAR files in the classpath.

public bool? ExtraJarsFirst { get; set; }

Property Value

bool?

Remarks

Default: false - priority is not given to user-provided jars

See: --user-jars-first in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

GlueVersion

Glue Version The version of Glue to use to execute this job.

public GlueVersion? GlueVersion { get; set; }

Property Value

GlueVersion?

Remarks

Default: - determined by the job type: 4.0 for ETL and Streaming, 5.0 for Flex, 3.0 for Python Shell

JobName

Name of the Glue job (optional) Developer-specified name of the Glue job.

public string? JobName { get; set; }

Property Value

string

Remarks

Default: - a name is automatically generated

MaxConcurrentRuns

Max Concurrent Runs (optional) The maximum number of runs this Glue job can concurrently run.

public double? MaxConcurrentRuns { get; set; }

Property Value

double?

Remarks

An error is returned when this threshold is reached. The maximum value you can specify is controlled by a service limit.

Default: 1

MaxRetries

Max Retries (optional) Maximum number of retry attempts Glue performs if the job fails.

public double? MaxRetries { get; set; }

Property Value

double?

Remarks

Default: 0

NotifyDelayAfter

Specifies configuration properties of a notification (optional).

public Duration? NotifyDelayAfter { get; set; }

Property Value

Duration

Remarks

After a job run starts, the number of minutes to wait before sending a job run delay notification.

Default: - undefined

Role

IAM Role (required) IAM Role to use for Glue job execution Must be specified by the developer because the L2 doesn't have visibility into the actions the script(s) takes during the job execution The role must trust the Glue service principal (glue.amazonaws.com) and be granted sufficient permissions.

public IRole Role { get; set; }

Property Value

IRole

Remarks

See: https://docs.aws.amazon.com/glue/latest/dg/getting-started-access.html

Script

Script Code Location (required) Script to run when the Glue job executes.

public Code Script { get; set; }

Property Value

Code

Remarks

Can be uploaded from the local directory structure using fromAsset or referenced via S3 location using fromBucket

SecurityConfiguration

Security Configuration (optional) Defines the encryption options for the Glue job.

public ISecurityConfiguration? SecurityConfiguration { get; set; }

Property Value

ISecurityConfiguration

Remarks

Default: - no security configuration.

SparkUI

Enables the Spark UI debugging and monitoring with the specified props.

public ISparkUIProps? SparkUI { get; set; }

Property Value

ISparkUIProps

Remarks

Default: - Spark UI debugging and monitoring is disabled.

See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

Tags

Tags (optional) A list of key:value pairs of tags to apply to this Glue job resources.

public IDictionary<string, string>? Tags { get; set; }

Property Value

IDictionary<string, string>

Remarks

Default: {} - no tags

Timeout

Timeout (optional) The maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status.

public Duration? Timeout { get; set; }

Property Value

Duration

Remarks

Specified in minutes.

Default: - no value set; Glue applies its service default (2880 minutes for non-streaming jobs)

WorkerConfiguration

The worker type and the number of workers allocated when a job runs.

public IWorkerConfiguration? WorkerConfiguration { get; set; }

Property Value

IWorkerConfiguration

Remarks

Default: - the job runs with the G_1X worker type and 10 workers.

Implements

IScalaSparkFlexEtlJobProps

ISparkJobProps

IJobProps