Streaming ETL jobs in AWS Glue

• Notes and restrictions for Avro streaming sources

• Applying grok patterns to streaming sources

To set streaming ETL properties for Amazon Kinesis Data Streams (AWS Glue API or AWS CLI)

• To set up streaming ETL properties for a Kinesis source in the same account, specify the streamName and endpointUrl parameters in the StorageDescriptor structure of the CreateTable API operation or the create_table CLI command.

  "StorageDescriptor": {
  	"Parameters": {
  		"typeOfData": "kinesis",
  		"streamName": "sample-stream",
  		"endpointUrl": "https://kinesis.us-east-1.amazonaws.com"
  	}
  	...
  }
  					

  Or, specify the streamARN.

  "StorageDescriptor": {
  	"Parameters": {
  		"typeOfData": "kinesis",
  		"streamARN": "arn:aws:kinesis:us-east-1:123456789:stream/sample-stream"
  	}
  	...
  }

• To set up streaming ETL properties for a Kinesis source in another account, specify the streamARN, awsSTSRoleARN and awsSTSSessionName (optional) parameters in the StorageDescriptor structure in the CreateTable API operation or the create_table CLI command.

  "StorageDescriptor": {
  	"Parameters": {
  		"typeOfData": "kinesis",
  		"streamARN": "arn:aws:kinesis:us-east-1:123456789:stream/sample-stream",
  		"awsSTSRoleARN": "arn:aws:iam::123456789:role/sample-assume-role-arn",
  		"awsSTSSessionName": "optional-session"
  	}
  	...
  }

• When schema detection is turned on, the Avro schema must be included in the payload. When turned off, the payload should contain only data.

• Some Avro data types are not supported in dynamic frames. You can't specify these data types when defining the schema with the Define a schema page in the create table wizard in the AWS Glue console. During schema detection, unsupported types in the Avro schema are converted to supported types as follows:

  • EnumType => StringType

  • FixedType => BinaryType

  • UnionType => StructType

• If you define the table schema using the Define a schema page in the console, the implied root element type for the schema is record. If you want a root element type other than record, for example array or map, you can't specify the schema using the Define a schema page. Instead you must skip that page and specify the schema either as a table property or within the ETL script.

  • To specify the schema in the table properties, complete the create table wizard, edit the table details, and add a new key-value pair under Table properties. Use the key avroSchema, and enter a schema JSON object for the value, as shown in the following screenshot.

    

  • To specify the schema in the ETL script, modify the datasource0 assignment statement and add the avroSchema key to the additional_options argument, as shown in the following Python and Scala examples.

    Python

    SCHEMA_STRING = ‘{"type":"array","items":"string"}’
    datasource0 = glueContext.create_data_frame.from_catalog(database = "database", table_name = "table_name", transformation_ctx = "datasource0", additional_options = {"startingPosition": "TRIM_HORIZON", "inferSchema": "false", "avroSchema": SCHEMA_STRING})
    

    Scala

    val SCHEMA_STRING = """{"type":"array","items":"string"}"""
    val datasource0 = glueContext.getCatalogSource(database = "database", tableName = "table_name", redshiftTmpDir = "", transformationContext = "datasource0", additionalOptions = JsonOptions(s"""{"startingPosition": "TRIM_HORIZON", "inferSchema": "false", "avroSchema":"$SCHEMA_STRING"}""")).getDataFrame()
    

• Auto-decompression for AWS Glue streaming ETL jobs is only available for the supported compression types. Also note the following:

  • Framed Snappy refers to the official framing format for Snappy.

  • Deflate is supported in Glue version 3.0, not Glue version 2.0.

• When using schema detection, you cannot perform joins of streaming data.

• AWS Glue streaming ETL jobs do not support the Union data type for AWS Glue Schema Registry with Avro format.

• Your ETL script can use AWS Glue's built-in transforms and the transforms native to Apache Spark Structured Streaming. For more information, see Operations on streaming DataFrames/Datasets on the Apache Spark website or AWS Glue PySpark transforms reference.

• AWS Glue streaming ETL jobs use checkpoints to keep track of the data that has been read. Therefore, a stopped and restarted job picks up where it left off in the stream. If you want to reprocess data, you can delete the checkpoint folder referenced in the script.

• Job bookmarks aren't supported.

• To use enhanced fan-out feature of Kinesis Data Streams in your job, consult Using enhanced fan-out in Kinesis streaming jobs.

• If you use a Data Catalog table created from AWS Glue Schema Registry, when a new schema version becomes available, to reflect the new schema, you need to do the following:

  1. Stop the jobs associated with the table.

  2. Update the schema for the Data Catalog table.

  3. Restart the jobs associated with the table.