Data lineage in Amazon DataZone

Understand the provenance of data: knowing where the data originated fosters trust in data by providing you with a clear understanding of its origins, dependencies, and transformations. This transparency helps in making confident data-driven decisions.

Understand the impact of changes to data pipelines: when changes are made to data pipelines, lineage can be used to identify all of the downstream consumers that are to be affected. This helps to ensure that changes are made without disrupting critical data flows.

Identify the root cause of data quality issues: if a data quality issue is detected in a downstream report, lineage, especially column-level lineage, can be used to trace the data back (at a column level) to identify the issue back to its source. This can help data engineers to identify and fix the problem.

Improve data governance and compliance: column-level lineage can be used to demonstrate compliance with data governance and privacy regulations. For example, column-level lineage can be used to show where sensitive data (such as PII) is stored and how it is processed in downstream activities.

Dataset node - this node type includes data lineage information about a specific data asset.

Job (run) node - this node type displays the details of the job, including the latest run of a particular job and run details. This node also captures multiple runs of the job and can be viewed in the History tab of the node details. You can view node details by choosing the node icon.

For dataset node with respective dataset type:

For jobs:

For job runs:

Column-level lineage: expand column-level lineage when available in dataset nodes. This automatically shows relationships with upstream or downstream dataset nodes if source column information is available.

Column search: when the default display for number of columns is 10. If there are more than 10 columns, pagination is activiated to navigate to the rest of the columns. To quickly view a particular column, you can search on the dataset node that list just the searched column.

View dataset nodes only: if you want to toggle to view only dataset lineage nodes and filter out the job nodes, you can choose the Open view control icon on the top left of the graph viewer and toggle the Display dataset nodes only option. This will remove all the job nodes from the graph and lets you navigate just the dataset nodes. Note that when the view only dataset nodes is turned on, the graph cannot be expanded upstream or downstream.

Details pane: Each lineage node has details captured and displayed when selected.

Version tabs: all lineage nodes in Amazon DataZone data lineage have versioning. For every dataset node or job node, the versions are captured as history and that enables you to navigate between the different versions to identify what has changed overtime. Each version opens a new tab in the lineage page to help compare or contrast.

GetLineageNode

ListLineageNodeHistory

PostLineageEvent

Blueprint configuration - administrators setting up blueprints can configure blueprints to capture lineage automatically. This enables the administrators to define which data sources are important for lineage capture rather than relying on data producers cataloguing data. For more information, see Enable data lineage in the management console.

Data source configuration - data producers, as they configure data source runs for AWS Glue databases, are presented with a view along with Data Quality to inform about automated data lineage for that data source.

AWS Glue (v5.0) configuration - while running AWS Glue jobs in AWS Glue Studio, data lineage can be configured for the jobs to send lineage events directly to Amazon DataZone domain.

AWS Glue (V5.0) Notebook configuration - in a notebook, you can automate the collection of Spark executions by adding a %%configure magic. This configuration will send events to Amazon DataZone domain.

%%configure
{
“—conf”:“spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener 
—conf spark.openlineage.transport.type=amazon_datazone_api 
—conf spark.openlineage.transport.domainId=<datazone domainID> 
—conf spark.openlineage.facets.custom_environment_variables [AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;] 
—conf spark.glue.JobName=<SessionId> 
—conf spark.glue.JobRunId=<SessionId or NONE?>“ (as session is a resource and doesn't have subsequent runs - interactive)
                    

Note: there are 2 dashes in front of conf - quip is updating to hyphen.

Set up parameters to configure communication to Amazon DataZone from AWS Glue

Param key: --conf

Param value:

spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener 
--conf spark.openlineage.transport.type=amazon_datazone_api 
--conf spark.openlineage.transport.domainId=<DOMAIN_ID>
--conf spark.openlineage.facets.custom_environment_variables=[AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;] 
--conf spark.glue.accountId=<ACCOUNT_ID> (replace <DOMAIN_ID> and <ACCOUNT_ID> with the right values)
                    

For Notebooks add these additinal parameters:

--conf spark.glue.JobName=<SessionId> --conf spark.glue.JobRunId=<SessionId or NONE?>
replace <SessionId> and <SessionId> with the right values
                    

Blueprint configuration: administrators setting up blueprints can configure blueprints to capture lineage automatically. This enables the administrators to define which data sources are important for lineage capture rather than relying on data producers cataloguing data. To setup, go to Enable data lineage in the management console.

Data source configuration: data producers, as they configure data source runs for Amazon Redshift databases, are presented with automated data lineage setting for that data source.

The lineage setting can be viewed in the Data Source Definition tab. This value is not editable by data producers.