Glue / Client / get_data_quality_result_v2

get_data_quality_result_v2¶

Glue.Client.get_data_quality_result_v2(**kwargs)¶

Retrieves the result of a data quality rule evaluation published from any supported data platform.

Request Syntax

response = client.get_data_quality_result_v2(
    ResultId='string'
)

Parameters:

ResultId (string) –

[REQUIRED]

A unique result ID for the data quality result.

Return type:

dict

Returns:

Response Syntax

{
    'ResultId': 'string',
    'DataSource': {
        'GlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        },
        'DataQualityGlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            },
            'PreProcessingQuery': 'string'
        }
    },
    'ProfileId': 'string',
    'Dimensions': {
        'string': 'string'
    },
    'Tags': {
        'string': 'string'
    },
    'ExecutionStartedOn': datetime(2015, 1, 1),
    'ExecutionCompletedOn': datetime(2015, 1, 1),
    'ResultPublishedOn': datetime(2015, 1, 1),
    'Score': 123.0,
    'RuleResults': [
        {
            'Name': 'string',
            'Description': 'string',
            'EvaluationMessage': 'string',
            'Result': 'PASS'|'FAIL'|'ERROR',
            'EvaluatedMetrics': {
                'string': 123.0
            },
            'EvaluatedRule': 'string',
            'RuleMetrics': {
                'string': 123.0
            },
            'Labels': {
                'string': 'string'
            }
        },
    ],
    'AnalyzerResults': [
        {
            'Name': 'string',
            'Description': 'string',
            'EvaluationMessage': 'string',
            'EvaluatedMetrics': {
                'string': 123.0
            },
            'EvaluatedDistributions': {
                'string': {
                    'BinEdges': [
                        'string',
                    ],
                    'Count': [
                        123,
                    ],
                    'DataType': 'string'
                }
            }
        },
    ],
    'Observations': [
        {
            'Description': 'string',
            'MetricBasedObservation': {
                'MetricName': 'string',
                'StatisticId': 'string',
                'MetricValues': {
                    'ActualValue': 123.0,
                    'ExpectedValue': 123.0,
                    'LowerLimit': 123.0,
                    'UpperLimit': 123.0
                },
                'NewRules': [
                    'string',
                ]
            }
        },
    ],
    'EvaluationContext': 'string',
    'Descriptors': {
        'string': 'string'
    },
    'AggregatedMetrics': {
        'TotalRowsProcessed': 123.0,
        'TotalRowsPassed': 123.0,
        'TotalRowsFailed': 123.0,
        'TotalRulesProcessed': 123.0,
        'TotalRulesPassed': 123.0,
        'TotalRulesFailed': 123.0
    },
    'RulesetName': 'string'
}

Response Structure

(dict) –
- ResultId (string) –
  
  A unique result ID for the data quality result.
- DataSource (dict) –
  
  The table associated with the data quality result, if any.
  - GlueTable (dict) –
    
    An Glue table.
    - DatabaseName (string) –
      
      A database name in the Glue Data Catalog.
    - TableName (string) –
      
      A table name in the Glue Data Catalog.
    - CatalogId (string) –
      
      A unique identifier for the Glue Data Catalog.
    - ConnectionName (string) –
      
      The name of the connection to the Glue Data Catalog.
    - AdditionalOptions (dict) –
      
      Additional options for the table. Currently there are two keys supported:
      - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
      - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
      - (string) –
        
        (string) –
  - DataQualityGlueTable (dict) –
    
    An Glue table for Data Quality Operations.
    - DatabaseName (string) –
      
      A database name in the Glue Data Catalog.
    - TableName (string) –
      
      A table name in the Glue Data Catalog.
    - CatalogId (string) –
      
      A unique identifier for the Glue Data Catalog.
    - ConnectionName (string) –
      
      The name of the connection to the Glue Data Catalog.
    - AdditionalOptions (dict) –
      
      Additional options for the table. Currently there are two keys supported:
      - pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.
      - catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.
      - (string) –
        
        (string) –
    - PreProcessingQuery (string) –
      
      SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.
- ProfileId (string) –
  
  The Profile ID for the data quality result.
- Dimensions (dict) –
  
  A map of platform-specific identifiers, such as cluster, application, workgroup, or project IDs, that identify where the data quality evaluation ran.
  - (string) –
    - (string) –
- Tags (dict) –
  
  A map of tags associated with the data quality result.
  - (string) –
    - (string) –
- ExecutionStartedOn (datetime) –
  
  The date and time when this data quality run started.
- ExecutionCompletedOn (datetime) –
  
  The date and time when this data quality run completed.
- ResultPublishedOn (datetime) –
  
  The date and time when this data quality result was published.
- Score (float) –
  
  An aggregate data quality score. Represents the ratio of rules that passed to the total number of rules.
- RuleResults (list) –
  
  A list of DataQualityRuleResult objects representing the results for each rule.
  - (dict) –
    
    Describes the result of the evaluation of a data quality rule.
    - Name (string) –
      
      The name of the data quality rule.
    - Description (string) –
      
      A description of the data quality rule.
    - EvaluationMessage (string) –
      
      An evaluation message.
    - Result (string) –
      
      A pass or fail status for the rule.
    - EvaluatedMetrics (dict) –
      
      A map of metrics associated with the evaluation of the rule.
      - (string) –
        
        (float) –
    - EvaluatedRule (string) –
      
      The evaluated rule.
    - RuleMetrics (dict) –
      
      A map containing metrics associated with the evaluation of the rule based on row-level results.
      - (string) –
        
        (float) –
    - Labels (dict) –
      
      A map containing labels assigned to the data quality rule.
      - (string) –
        
        (string) –
- AnalyzerResults (list) –
  
  A list of DataQualityAnalyzerResult objects representing the results for each analyzer.
  - (dict) –
    
    Describes the result of the evaluation of a data quality analyzer.
    - Name (string) –
      
      The name of the data quality analyzer.
    - Description (string) –
      
      A description of the data quality analyzer.
    - EvaluationMessage (string) –
      
      An evaluation message.
    - EvaluatedMetrics (dict) –
      
      A map of metrics associated with the evaluation of the analyzer.
      - (string) –
        
        (float) –
    - EvaluatedDistributions (dict) –
      
      A map of distribution metrics associated with the evaluation of the analyzer.
      - (string) –
        
        (dict) –
        
        The distribution data for a statistic.
        
        BinEdges (list) –
        
        The bin edge values for the distribution.
        
        (string) –
        
        Count (list) –
        
        The frequency count for each bin in the distribution.
        
        (integer) –
        
        DataType (string) –
        
        The data type of the column for the distribution.
- Observations (list) –
  
  A list of DataQualityObservation objects representing the anomaly detection observations generated after evaluating the rules and analyzers.
  - (dict) –
    
    Describes the observation generated after evaluating the rules and analyzers.
    - Description (string) –
      
      A description of the data quality observation.
    - MetricBasedObservation (dict) –
      
      An object of type MetricBasedObservation representing the observation that is based on evaluated data quality metrics.
      - MetricName (string) –
        
        The name of the data quality metric used for generating the observation.
      - StatisticId (string) –
        
        The Statistic ID.
      - MetricValues (dict) –
        
        An object of type DataQualityMetricValues representing the analysis of the data quality metric value.
        
        ActualValue (float) –
        
        The actual value of the data quality metric.
        
        ExpectedValue (float) –
        
        The expected value of the data quality metric according to the analysis of historical data.
        
        LowerLimit (float) –
        
        The lower limit of the data quality metric value according to the analysis of historical data.
        
        UpperLimit (float) –
        
        The upper limit of the data quality metric value according to the analysis of historical data.
      - NewRules (list) –
        
        A list of new data quality rules generated as part of the observation based on the data quality metric value.
        
        (string) –
- EvaluationContext (string) –
  
  The evaluation context for the data quality result. This value identifies a specific data quality result, or a series of related results, within a run.
- Descriptors (dict) –
  
  A map of user-provided descriptors that add contextual metadata and configuration to the data quality result, such as database name, table name, team, or environment.
  - (string) –
    - (string) –
- AggregatedMetrics (dict) –
  
  A summary of DataQualityAggregatedMetrics objects showing the total counts of processed rows and rules, including their pass/fail statistics based on row-level results.
  - TotalRowsProcessed (float) –
    
    The total number of rows that were processed during the data quality evaluation.
  - TotalRowsPassed (float) –
    
    The total number of rows that passed all applicable data quality rules.
  - TotalRowsFailed (float) –
    
    The total number of rows that failed one or more data quality rules.
  - TotalRulesProcessed (float) –
    
    The total number of data quality rules that were evaluated.
  - TotalRulesPassed (float) –
    
    The total number of data quality rules that passed their evaluation criteria.
  - TotalRulesFailed (float) –
    
    The total number of data quality rules that failed their evaluation criteria.
- RulesetName (string) –
  
  The name of the ruleset associated with the data quality result.

Exceptions

Glue.Client.exceptions.InvalidInputException
Glue.Client.exceptions.OperationTimeoutException
Glue.Client.exceptions.InternalServiceException
Glue.Client.exceptions.EntityNotFoundException