Glue / Client / batch_get_data_quality_result_v2

batch_get_data_quality_result_v2

Glue.Client.batch_get_data_quality_result_v2(**kwargs)

Retrieves a list of data quality results for the specified result IDs. You can specify up to 100 result IDs in a single request.

See also: AWS API Documentation

Request Syntax

response = client.batch_get_data_quality_result_v2(
    ResultIds=[
        'string',
    ]
)

Parameters:

ResultIds (list) –

[REQUIRED]

A list of unique result IDs for the data quality results.

• (string) –

Return type:

dict

Returns:

Response Syntax

{
    'Results': [
        {
            'ResultId': 'string',
            'DataSource': {
                'GlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    }
                },
                'DataQualityGlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'PreProcessingQuery': 'string'
                }
            },
            'Dimensions': {
                'string': 'string'
            },
            'Tags': {
                'string': 'string'
            },
            'ExecutionStartedOn': datetime(2015, 1, 1),
            'ExecutionCompletedOn': datetime(2015, 1, 1),
            'ResultPublishedOn': datetime(2015, 1, 1),
            'Score': 123.0,
            'RuleResults': [
                {
                    'Name': 'string',
                    'Description': 'string',
                    'EvaluationMessage': 'string',
                    'Result': 'PASS'|'FAIL'|'ERROR',
                    'EvaluatedMetrics': {
                        'string': 123.0
                    },
                    'EvaluatedRule': 'string',
                    'RuleMetrics': {
                        'string': 123.0
                    },
                    'Labels': {
                        'string': 'string'
                    }
                },
            ],
            'AnalyzerResults': [
                {
                    'Name': 'string',
                    'Description': 'string',
                    'EvaluationMessage': 'string',
                    'EvaluatedMetrics': {
                        'string': 123.0
                    },
                    'EvaluatedDistributions': {
                        'string': {
                            'BinEdges': [
                                'string',
                            ],
                            'Count': [
                                123,
                            ],
                            'DataType': 'string'
                        }
                    }
                },
            ],
            'Observations': [
                {
                    'Description': 'string',
                    'MetricBasedObservation': {
                        'MetricName': 'string',
                        'StatisticId': 'string',
                        'MetricValues': {
                            'ActualValue': 123.0,
                            'ExpectedValue': 123.0,
                            'LowerLimit': 123.0,
                            'UpperLimit': 123.0
                        },
                        'NewRules': [
                            'string',
                        ]
                    }
                },
            ],
            'EvaluationContext': 'string',
            'Descriptors': {
                'string': 'string'
            },
            'AggregatedMetrics': {
                'TotalRowsProcessed': 123.0,
                'TotalRowsPassed': 123.0,
                'TotalRowsFailed': 123.0,
                'TotalRulesProcessed': 123.0,
                'TotalRulesPassed': 123.0,
                'TotalRulesFailed': 123.0
            },
            'RulesetName': 'string'
        },
    ],
    'ResultsNotFound': [
        'string',
    ]
}

Response Structure

• (dict) –

  • Results (list) –

    A list of DataQualityResultV2Item objects representing the data quality results.

    • (dict) –

      Describes a data quality result returned by BatchGetDataQualityResultV2.

      • ResultId (string) –

        A unique result ID for the data quality result.

      • DataSource (dict) –

        The table associated with the data quality result, if any.

        • GlueTable (dict) –

          An Glue table.

          • DatabaseName (string) –

            A database name in the Glue Data Catalog.

          • TableName (string) –

            A table name in the Glue Data Catalog.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

          • ConnectionName (string) –

            The name of the connection to the Glue Data Catalog.

          • AdditionalOptions (dict) –

            Additional options for the table. Currently there are two keys supported:

            • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

            • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

            • (string) –

              • (string) –

        • DataQualityGlueTable (dict) –

          An Glue table for Data Quality Operations.

          • DatabaseName (string) –

            A database name in the Glue Data Catalog.

          • TableName (string) –

            A table name in the Glue Data Catalog.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

          • ConnectionName (string) –

            The name of the connection to the Glue Data Catalog.

          • AdditionalOptions (dict) –

            Additional options for the table. Currently there are two keys supported:

            • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

            • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

            • (string) –

              • (string) –

          • PreProcessingQuery (string) –

            SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.

      • Dimensions (dict) –

        A map of platform-specific identifiers, such as cluster, application, workgroup, or project IDs, that identify where the data quality evaluation ran.

        • (string) –

          • (string) –

      • Tags (dict) –

        A map of tags associated with the data quality result.

        • (string) –

          • (string) –

      • ExecutionStartedOn (datetime) –

        The date and time when this data quality run started.

      • ExecutionCompletedOn (datetime) –

        The date and time when this data quality run completed.

      • ResultPublishedOn (datetime) –

        The date and time when this data quality result was published.

      • Score (float) –

        An aggregate data quality score. Represents the ratio of rules that passed to the total number of rules.

      • RuleResults (list) –

        A list of DataQualityRuleResult objects representing the results for each rule.

        • (dict) –

          Describes the result of the evaluation of a data quality rule.

          • Name (string) –

            The name of the data quality rule.

          • Description (string) –

            A description of the data quality rule.

          • EvaluationMessage (string) –

            An evaluation message.

          • Result (string) –

            A pass or fail status for the rule.

          • EvaluatedMetrics (dict) –

            A map of metrics associated with the evaluation of the rule.

            • (string) –

              • (float) –

          • EvaluatedRule (string) –

            The evaluated rule.

          • RuleMetrics (dict) –

            A map containing metrics associated with the evaluation of the rule based on row-level results.

            • (string) –

              • (float) –

          • Labels (dict) –

            A map containing labels assigned to the data quality rule.

            • (string) –

              • (string) –

      • AnalyzerResults (list) –

        A list of DataQualityAnalyzerResult objects representing the results for each analyzer.

        • (dict) –

          Describes the result of the evaluation of a data quality analyzer.

          • Name (string) –

            The name of the data quality analyzer.

          • Description (string) –

            A description of the data quality analyzer.

          • EvaluationMessage (string) –

            An evaluation message.

          • EvaluatedMetrics (dict) –

            A map of metrics associated with the evaluation of the analyzer.

            • (string) –

              • (float) –

          • EvaluatedDistributions (dict) –

            A map of distribution metrics associated with the evaluation of the analyzer.

            • (string) –

              • (dict) –

                The distribution data for a statistic.

                • BinEdges (list) –

                  The bin edge values for the distribution.

                  • (string) –

                • Count (list) –

                  The frequency count for each bin in the distribution.

                  • (integer) –

                • DataType (string) –

                  The data type of the column for the distribution.

      • Observations (list) –

        A list of DataQualityObservation objects representing the anomaly detection observations generated after evaluating the rules and analyzers.

        • (dict) –

          Describes the observation generated after evaluating the rules and analyzers.

          • Description (string) –

            A description of the data quality observation.

          • MetricBasedObservation (dict) –

            An object of type MetricBasedObservation representing the observation that is based on evaluated data quality metrics.

            • MetricName (string) –

              The name of the data quality metric used for generating the observation.

            • StatisticId (string) –

              The Statistic ID.

            • MetricValues (dict) –

              An object of type DataQualityMetricValues representing the analysis of the data quality metric value.

              • ActualValue (float) –

                The actual value of the data quality metric.

              • ExpectedValue (float) –

                The expected value of the data quality metric according to the analysis of historical data.

              • LowerLimit (float) –

                The lower limit of the data quality metric value according to the analysis of historical data.

              • UpperLimit (float) –

                The upper limit of the data quality metric value according to the analysis of historical data.

            • NewRules (list) –

              A list of new data quality rules generated as part of the observation based on the data quality metric value.

              • (string) –

      • EvaluationContext (string) –

        The evaluation context for the data quality result. This value identifies a specific data quality result, or a series of related results, within a run.

      • Descriptors (dict) –

        A map of user-provided descriptors that add contextual metadata and configuration to the data quality result, such as database name, table name, team, or environment.

        • (string) –

          • (string) –

      • AggregatedMetrics (dict) –

        A summary of DataQualityAggregatedMetrics objects showing the total counts of processed rows and rules, including their pass/fail statistics based on row-level results.

        • TotalRowsProcessed (float) –

          The total number of rows that were processed during the data quality evaluation.

        • TotalRowsPassed (float) –

          The total number of rows that passed all applicable data quality rules.

        • TotalRowsFailed (float) –

          The total number of rows that failed one or more data quality rules.

        • TotalRulesProcessed (float) –

          The total number of data quality rules that were evaluated.

        • TotalRulesPassed (float) –

          The total number of data quality rules that passed their evaluation criteria.

        • TotalRulesFailed (float) –

          The total number of data quality rules that failed their evaluation criteria.

      • RulesetName (string) –

        The name of the ruleset associated with the data quality result.

  • ResultsNotFound (list) –

    A list of result IDs for which results were not found.

    • (string) –

Exceptions

• Glue.Client.exceptions.InvalidInputException

• Glue.Client.exceptions.OperationTimeoutException

• Glue.Client.exceptions.InternalServiceException