Reading from a REST API

After you register a REST API ConnectionType and create an AWS Glue connection, you can read data from the REST API in your AWS Glue ETL jobs. With this connection, you can process external REST API data alongside other sources in the same job. You need the connection name and entity name to read data.

The following example shows how to read from a REST API data source using Python:

rest_read = glueContext.create_dynamic_frame.from_options(
    connection_type="rest",
    connection_options={
        "connectionName": "connection-name",
        "ENTITY_NAME": "entity-name",
        "CONNECTION_TYPE": "REST-connection-type"
    }
)

Filtering data

You can push down filter predicates to the source REST API to reduce the amount of data transferred. The REST API connector supports two filter modes:

To apply a filter predicate in your AWS Glue ETL job, use the FILTER_PREDICATE connection option:

rest_read = glueContext.create_dynamic_frame.from_options(
    connection_type="rest",
    connection_options={
        "connectionName": "connection-name",
        "ENTITY_NAME": "entity-name",
        "CONNECTION_TYPE": "REST-connection-type",
        "FILTER_PREDICATE": "status = \"ACTIVE\" AND lastUpdated >= \"2024-01-01T00:00:00.000Z\""
    }
)

FilterConfiguration properties

Configure filtering behavior in your ConnectionType registration using the FilterConfiguration object. The following table describes the available properties:

FilterStringConfiguration properties

The following table describes the properties for FILTER_STRING mode:

BetweenConfiguration properties

The following table describes the BETWEEN handling properties:

Supported operators

The following operators are supported in filter predicates: EQUAL_TO, GREATER_THAN, LESS_THAN, GREATER_THAN_OR_EQUAL_TO, LESS_THAN_OR_EQUAL_TO, NOT_EQUAL_TO, CONTAINS, BETWEEN, AND, OR.

Field-level overrides

You can configure per-field filter behavior using FilterOverrides in the Schema definition. The following override properties are available:

Example: QUERY_PARAMS mode

The following example shows a FilterConfiguration for a REST API that uses query parameters with bracket notation for operators and epoch timestamps for date values:

"FilterConfiguration": {
    "FilterMode": "QUERY_PARAMS",
    "OperatorMappings": {
        "EQUAL_TO": "{FIELD}",
        "GREATER_THAN_OR_EQUAL_TO": "{FIELD}[gte]",
        "LESS_THAN_OR_EQUAL_TO": "{FIELD}[lte]"
    },
    "DateTimeFormat": "EPOCH_SECONDS",
    "BetweenConfiguration": {
        "LowBoundKey": "{FIELD}[gte]",
        "HighBoundKey": "{FIELD}[lte]"
    }
}

With this configuration, an input filter of created >= 2024-01-01 AND created <= 2024-06-01 produces the URL query string: ?created[gte]=1704067200&created[lte]=1717200000

Example: FILTER_STRING mode

The following example shows a FilterConfiguration for a REST API that uses a single filter string parameter with space-padded operators:

"FilterConfiguration": {
    "FilterMode": "FILTER_STRING",
    "OperatorMappings": {
        "EQUAL_TO": " eq ",
        "GREATER_THAN": " gt ",
        "GREATER_THAN_OR_EQUAL_TO": " ge ",
        "LESS_THAN": " lt ",
        "AND": " and ",
        "OR": " or "
    },
    "DateTimeFormat": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
    "BetweenConfiguration": {
        "Template": "{FIELD} ge {LOW} and {FIELD} le {HIGH}"
    },
    "FilterStringConfiguration": {
        "FilterStringKey": "search",
        "QuoteStringValues": true,
        "QuoteCharacter": "\""
    }
}

With this configuration, an input filter of status = "ACTIVE" AND lastUpdated > 2024-01-01T00:00:00.000Z produces the URL query string: ?search=status eq "ACTIVE" and lastUpdated gt "2024-01-01T00:00:00.000Z"

Partitioning queries

You can split data reads into parallel partitions across Spark workers to improve throughput. The REST API connector supports field-based partitioning, which divides data into ranges based on a specified field.

Spark job parameters

The following connection options control partitioning behavior:

Field-based partitioning

Field-based partitioning divides data into ranges based on a specified field. The following example partitions data on the lastUpdated field:

rest_read = glueContext.create_dynamic_frame.from_options(
    connection_type="rest",
    connection_options={
        "connectionName": "connection-name",
        "ENTITY_NAME": "entity-name",
        "CONNECTION_TYPE": "REST-connection-type",
        "PARTITION_FIELD": "lastUpdated",
        "LOWER_BOUND": "2024-01-01T00:00:00.000Z",
        "UPPER_BOUND": "2024-12-31T00:00:00.000Z",
        "NUM_PARTITIONS": "4"
    }
)

With 4 partitions on the lastUpdated field, the work is distributed as follows:

Configuring partition support in RegisterConnectionType

To enable field-based partitioning, mark fields as partitionable in the Schema definition:

"Schema": {
    "lastUpdated": {
        "FieldDataType": "TIMESTAMP",
        "IsPartitionable": true
    }
}

The following FieldDataType values are supported for partitioning: