Troubleshooting Amazon DataZone

If you encounter access-denied issues or similar difficulties when working with Amazon DataZone consult the topics in this section.

Troubleshooting AWS Lake Formation permissions for Amazon DataZone

This section contains troubleshooting instructions for issues that you might encounter when you Configure Lake Formation permissions for Amazon DataZone.

Troubleshooting Amazon DataZone lineage asset linking with upstream datasets

This section contains troubleshooting instructions for issues that you might encounter with Amazon DataZone lineage. For some of the AWS Glue and Amazon Redshift-related open lineage run events, you may see that asset lineage is not linked to an upstream dataset. This topic explains the scenarios and a few approaches to mitigate issues. For more information on lineage, see Data lineage in Amazon DataZone.

SourceIdentifier on lineage node

The sourceIdentifier attribute in a lineage node represents the events happening on a dataset. For more information, see Key attributes in lineage nodes.

The lineage node represents all the events that happen on the corresponding dataset or job. The lineage node contains a "sourceIdentifier" attribute which contains the identifier of the corresponding dataset/job. As we support open-lineage events, the sourceIdentifier value is by default populated as the combination of "namespace" and "name" for a dataset, job and job runs.

For AWS resources such as AWS Glue and Amazon Redshift, the sourceIdentifier would be the AWS Glue table ARN and the Redshift table ARNs from which Amazon DataZone will construct the run-event and other details as follows:

SourceIdentifier on the assets within DataZone

AssetCommonDetailForm has an attribute called "sourceIdentifier" which represents the identifier of the dataset which the asset represents. For asset lineage nodes to be linked with an upstream dataset, the attribute needs to be populated with the value matching with the dataset node’s sourceIdentifier. If the assets are imported by datasource, the workflow populates sourceIdentifier as the AWS Glue table ARN / Redshift table ARN automatically while other assets (including custom assets) created via the CreateAsset API should have that value populated by the caller.

How does Amazon DataZone construct the sourceIdentifier from the OpenLineage Event?

For AWS Glue and Redshift assets, the sourceIdentifier is constructed from Glue and Redshift ARNs. Here's how Amazon DataZone constructs it:

AWS Glue ARN

The goal is to construct an OpenLineage Event where the output lineage node's sourceIdentifier is:

arn:aws:glue:us-east-1:123456789012:table/testlfdb/testlftb-1

To determine if a run is using data from AWS Glue, look for the presence of certain keywords in the environment-properties facet. Specifically, if any of these designated fields are present, the system assumes the RunEvent originates from AWS Glue.

"run": {
   "runId":"4e3da9e8-6228-4679-b0a2-fa916119fthr",
   "facets":{
      "environment-properties":{
         "_producer":"https://github.com/OpenLineage/OpenLineage/tree/1.9.1/integration/spark",
         "_schemaURL":"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet",
         "environment-properties":{
            "GLUE_VERSION":"3.0",
            "GLUE_COMMAND_CRITERIA":"glueetl",
            "GLUE_PYTHON_VERSION":"3"
         }
      }
   }

For an AWS Glue run, you can use the name from the symlinks facet to get the database and table name, which can be used to construct the ARN.

Need to make sure the name is databaseName.tableName:

"symlinks": {
   "_producer":"https://github.com/OpenLineage/OpenLineage/tree/1.9.1/integration/spark",
   "_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet",
   "identifiers":[
      {
         "namespace":"s3://object-path",
         "name":"testlfdb.testlftb-1",
         "type":"TABLE"
      }
   ]
}

Sample COMPLETE Event:

{
   "eventTime":"2024-07-01T12:00:00.000000Z",
   "producer":"https://github.com/OpenLineage/OpenLineage/tree/1.9.1/integration/glue",
   "schemaURL":"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
   "eventType":"COMPLETE",
   "run": {
      "runId":"4e3da9e8-6228-4679-b0a2-fa916119fthr",
      "facets":{
         "environment-properties":{
            "_producer":"https://github.com/OpenLineage/OpenLineage/tree/1.9.1/integration/spark",
            "_schemaURL":"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet",
            "environment-properties":{
               "GLUE_VERSION":"3.0",
               "GLUE_COMMAND_CRITERIA":"glueetl",
               "GLUE_PYTHON_VERSION":"3"
            }
         }
      }
   },
   "job":{
      "namespace":"namespace",
      "name":"job_name",
      "facets":{
         "jobType":{
            "_producer":"https://github.com/OpenLineage/OpenLineage/tree/1.9.1/integration/glue",
            "_schemaURL":"https://openlineage.io/spec/facets/2-0-2/JobTypeJobFacet.json#/$defs/JobTypeJobFacet",
            "processingType":"BATCH",
            "integration":"glue",
            "jobType":"JOB"
         }
      }
   },
   "inputs":[
      {
         "namespace":"namespace",
         "name":"input_name"
      }
   ],
   "outputs":[
      {
         "namespace":"namespace.output",
         "name":"output_name",
         "facets":{
            "symlinks":{
               "_producer":"https://github.com/OpenLineage/OpenLineage/tree/1.9.1/integration/spark",
               "_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet",
               "identifiers":[
                  {
                     "namespace":"s3://object-path",
                     "name":"testlfdb.testlftb-1",
                     "type":"TABLE"
                  }
               ]
            }
         }
      }
   ]
}

Based on the OpenLineage event submitted, the sourceIdentifier of the output lineage node will be:

arn:aws:glue:us-east-1:123456789012:table/testlfdb/testlftb-1

The output lineage node will be connected to an asset's lineage node where the asset's sourceIdentifier is:

arn:aws:glue:us-east-1:123456789012:table/testlfdb/testlftb-1

Amazon Redshift ARN

The goal is to construct an OpenLineage Event where the output lineage node's sourceIdentifier is:

arn:aws:redshift:us-east-1:123456789012:table/workgroup-20240715/tpcds_data/public/dws_tpcds_7

The system determines whether an input or output is stored in Redshift based on the namespace. Specifically, if the namespace starts with redshift:// or contains the strings redshift-serverless.amazonaws.com or redshift.amazonaws.com, it is a Redshift resource.

"outputs": [
    {
        "namespace":"redshift://workgroup-20240715.123456789012.us-east-1.redshift.amazonaws.com:5439",
        "name":"tpcds_data.public.dws_tpcds_7"
    }
]

Note that the namespace needs to be in the following format:

provider://{cluster_identifier}.{region_name}:{port}

For redshift-serverless:

"outputs": [
    {
        "namespace":"redshift://workgroup-20240715.123456789012.us-east-1.redshift-serverless.amazonaws.com:5439",
        "name":"tpcds_data.public.dws_tpcds_7"
    }
]

Results in the following sourceIdentifier

arn:aws:redshift-serverless:us-east-1:123456789012:table/workgroup-20240715/tpcds_data/public/dws_tpcds_7

Based on the OpenLineage event submitted, the sourceIdentifier to be mapped to a downstream (that is, an output of the event) lineage node is:

arn:aws:redshift-serverless:us-e:us-east-1:123456789012:table/workgroup-20240715/tpcds_data/public/dws_tpcds_7

This is the mapping that helps you visualize the lineage of an asset in the catalog.

Alternate approach

When none of the above conditions are met, the system uses the namespace/name to construct the sourceIdentifier:

"inputs": [
  {
     "namespace":"arn:aws:redshift:us-east-1:123456789012:table",
     "name":"workgroup-20240715/tpcds_data/public/dws_tpcds_7"
  }
],
"outputs": [
  {
     "namespace":"arn:aws:glue:us-east-1:123456789012:table",
     "name":"testlfdb/testlftb-1"
  }
]

Troubleshooting a lack of upstream for the asset lineage node

If you don’t see the upstream of the asset lineage node, you can do the following to troubleshoot why it's not linked with the dataset:

Non-matching scenarios and mitigations

The following are commonly known scenarios where these will not match and the possible mitigations:

Root cause: The tables are present in different account than that of the Amazon DataZone domain account.

Mitigation: You can invoke the PostLineageEvent operation from an associated account. As the accountId to construct the ARN is picked from the caller credentials, you can assume the role from the account containing the tables when running the getting started script or invoking PostLineageEvent. Doing so will help in constructing the ARNs correctly and linking with the asset nodes.

Root cause: The ARN for Redshift table/views contains Redshift/Redshift-serverless based on the namespace and name attributes of the corresponding dataset information in the OpenLineage run event.

Mitigation: As there is no deterministic way to know if the given name belongs to cluster or workgroup, we use the following heuristic:

Root cause: Upstream datasets are not linked properly for custom assets.

Mitigation: Make sure to populate the sourceIdentifier on the asset by invoking CreateAsset/CreateAssetRevision that matches with the sourceIdentifier of the dataset node (which would be <namespace>/<name> for custom nodes).