Glue / Client / batch_get_data_quality_ruleset_evaluation_run

batch_get_data_quality_ruleset_evaluation_run

Glue.Client.batch_get_data_quality_ruleset_evaluation_run(**kwargs)

Retrieves the details of multiple evaluation runs in a single request.

See also: AWS API Documentation

Request Syntax

response = client.batch_get_data_quality_ruleset_evaluation_run(
    RunIds=[
        'string',
    ]
)

Parameters:

RunIds (list) –

[REQUIRED]

A list of unique run identifiers for the evaluation runs to retrieve.

• (string) –

Return type:

dict

Returns:

Response Syntax

{
    'Runs': [
        {
            'RunId': 'string',
            'DataSource': {
                'GlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    }
                },
                'DataQualityGlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'PreProcessingQuery': 'string'
                }
            },
            'Role': 'string',
            'NumberOfWorkers': 123,
            'Timeout': 123,
            'AdditionalRunOptions': {
                'CloudWatchMetricsEnabled': True|False,
                'ResultsS3Prefix': 'string',
                'CompositeRuleEvaluationMethod': 'COLUMN'|'ROW',
                'CustomLogGroupPrefix': 'string',
                'RowLevelResults': {
                    'MaxRowsToWrite': 123,
                    'ResultType': 'ALL'|'PASSED_ONLY'|'FAILED_ONLY',
                    'CatalogTableConfig': {
                        'DatabaseName': 'string',
                        'TableName': 'string',
                        'S3Location': 'string',
                        'CatalogId': 'string'
                    }
                },
                'ProfilingResults': {
                    'WriteProfilingResultsEnabled': True|False,
                    'CatalogTableConfig': {
                        'DatabaseName': 'string',
                        'TableName': 'string',
                        'S3Location': 'string',
                        'CatalogId': 'string'
                    },
                    'DistributionResults': {
                        'WriteDistributionResultsEnabled': True|False,
                        'CatalogTableConfig': {
                            'DatabaseName': 'string',
                            'TableName': 'string',
                            'S3Location': 'string',
                            'CatalogId': 'string'
                        }
                    }
                },
                'ObservationScope': 'ALL'|'NONE',
                'ObservationMode': 'SCHEDULED'|'FIXED',
                'DataQualityRuleResults': {
                    'WriteDataQualityRuleResultsEnabled': True|False,
                    'CatalogTableConfig': {
                        'DatabaseName': 'string',
                        'TableName': 'string',
                        'S3Location': 'string',
                        'CatalogId': 'string'
                    }
                },
                'ObservationResults': {
                    'WriteObservationResultsEnabled': True|False,
                    'CatalogTableConfig': {
                        'DatabaseName': 'string',
                        'TableName': 'string',
                        'S3Location': 'string',
                        'CatalogId': 'string'
                    }
                }
            },
            'Status': 'STARTING'|'RUNNING'|'STOPPING'|'STOPPED'|'SUCCEEDED'|'FAILED'|'TIMEOUT',
            'ErrorString': 'string',
            'StartedOn': datetime(2015, 1, 1),
            'LastModifiedOn': datetime(2015, 1, 1),
            'CompletedOn': datetime(2015, 1, 1),
            'ExecutionTime': 123,
            'RulesetNames': [
                'string',
            ],
            'ResultIds': [
                'string',
            ],
            'AdditionalDataSources': {
                'string': {
                    'GlueTable': {
                        'DatabaseName': 'string',
                        'TableName': 'string',
                        'CatalogId': 'string',
                        'ConnectionName': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        }
                    },
                    'DataQualityGlueTable': {
                        'DatabaseName': 'string',
                        'TableName': 'string',
                        'CatalogId': 'string',
                        'ConnectionName': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'PreProcessingQuery': 'string'
                    }
                }
            }
        },
    ],
    'RunsNotFound': [
        'string',
    ]
}

Response Structure

• (dict) –

  • Runs (list) –

    A list of evaluation run details for the requested run IDs.

    • (dict) –

      The details of a data quality ruleset evaluation run.

      • RunId (string) –

        The unique run identifier associated with this run.

      • DataSource (dict) –

        A data source (an Glue table) for which you want data quality results.

        • GlueTable (dict) –

          An Glue table.

          • DatabaseName (string) –

            A database name in the Glue Data Catalog.

          • TableName (string) –

            A table name in the Glue Data Catalog.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

          • ConnectionName (string) –

            The name of the connection to the Glue Data Catalog.

          • AdditionalOptions (dict) –

            Additional options for the table. Currently there are two keys supported:

            • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

            • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

            • (string) –

              • (string) –

        • DataQualityGlueTable (dict) –

          An Glue table for Data Quality Operations.

          • DatabaseName (string) –

            A database name in the Glue Data Catalog.

          • TableName (string) –

            A table name in the Glue Data Catalog.

          • CatalogId (string) –

            A unique identifier for the Glue Data Catalog.

          • ConnectionName (string) –

            The name of the connection to the Glue Data Catalog.

          • AdditionalOptions (dict) –

            Additional options for the table. Currently there are two keys supported:

            • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

            • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

            • (string) –

              • (string) –

          • PreProcessingQuery (string) –

            SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.

      • Role (string) –

        An IAM role supplied to encrypt the results of the run.

      • NumberOfWorkers (integer) –

        The number of G.1X workers to be used in the run. The default is 5.

      • Timeout (integer) –

        The timeout for a run in minutes. This is the maximum time that a run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).

      • AdditionalRunOptions (dict) –

        Additional run options you can specify for an evaluation run.

        • CloudWatchMetricsEnabled (boolean) –

          Whether or not to enable CloudWatch metrics.

        • ResultsS3Prefix (string) –

          Prefix for Amazon S3 to store results.

        • CompositeRuleEvaluationMethod (string) –

          Set the evaluation method for composite rules in the ruleset to ROW/COLUMN

        • CustomLogGroupPrefix (string) –

          A custom prefix for the CloudWatch log group names. When specified, evaluation run logs are written to <CustomLogGroupPrefix>/error and <CustomLogGroupPrefix>/output instead of the default /aws-glue/data-quality/error and /aws-glue/data-quality/output log groups.

        • RowLevelResults (dict) –

          The configuration for writing row-level evaluation results to a Glue Data Catalog table.

          • MaxRowsToWrite (integer) –

            The maximum number of rows to write in the results.

          • ResultType (string) –

            The result type to include in the row-level results output.

          • CatalogTableConfig (dict) –

            The Glue Data Catalog table configuration for storing the results.

            • DatabaseName (string) –

              The name of the database in the Glue Data Catalog.

            • TableName (string) –

              The name of the table in the Glue Data Catalog.

            • S3Location (string) –

              The Amazon S3 location for storing the results.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

        • ProfilingResults (dict) –

          The configuration for writing profiling results to a Glue Data Catalog table.

          • WriteProfilingResultsEnabled (boolean) –

            Set to true to write profiling results.

          • CatalogTableConfig (dict) –

            The Glue Data Catalog table configuration for storing the profiling results.

            • DatabaseName (string) –

              The name of the database in the Glue Data Catalog.

            • TableName (string) –

              The name of the table in the Glue Data Catalog.

            • S3Location (string) –

              The Amazon S3 location for storing the results.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

          • DistributionResults (dict) –

            The configuration for writing distribution results.

            • WriteDistributionResultsEnabled (boolean) –

              Set to true to write distribution results.

            • CatalogTableConfig (dict) –

              The Glue Data Catalog table configuration for storing the distribution results.

              • DatabaseName (string) –

                The name of the database in the Glue Data Catalog.

              • TableName (string) –

                The name of the table in the Glue Data Catalog.

              • S3Location (string) –

                The Amazon S3 location for storing the results.

              • CatalogId (string) –

                A unique identifier for the Glue Data Catalog.

        • ObservationScope (string) –

          The scope of the observation for the evaluation run. Specifies whether anomaly detection is enabled or disabled.

        • ObservationMode (string) –

          The observation mode for the evaluation run. Specifies how anomaly detection bounds are calculated.

        • DataQualityRuleResults (dict) –

          The configuration for writing rule results to a Glue Data Catalog table.

          • WriteDataQualityRuleResultsEnabled (boolean) –

            Set to true to write data quality rule results.

          • CatalogTableConfig (dict) –

            The Glue Data Catalog table configuration for storing the rule results.

            • DatabaseName (string) –

              The name of the database in the Glue Data Catalog.

            • TableName (string) –

              The name of the table in the Glue Data Catalog.

            • S3Location (string) –

              The Amazon S3 location for storing the results.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

        • ObservationResults (dict) –

          The configuration for writing observation results to a Glue Data Catalog table.

          • WriteObservationResultsEnabled (boolean) –

            Set to true to write observation results.

          • CatalogTableConfig (dict) –

            The Glue Data Catalog table configuration for storing the observation results.

            • DatabaseName (string) –

              The name of the database in the Glue Data Catalog.

            • TableName (string) –

              The name of the table in the Glue Data Catalog.

            • S3Location (string) –

              The Amazon S3 location for storing the results.

            • CatalogId (string) –

              A unique identifier for the Glue Data Catalog.

      • Status (string) –

        The status for this run.

      • ErrorString (string) –

        The error strings that are associated with the run.

      • StartedOn (datetime) –

        The date and time when this run started.

      • LastModifiedOn (datetime) –

        A timestamp. The last point in time when this run was modified.

      • CompletedOn (datetime) –

        The date and time when this run was completed.

      • ExecutionTime (integer) –

        The amount of time (in seconds) that the run consumed resources.

      • RulesetNames (list) –

        A list of ruleset names for the run.

        • (string) –

      • ResultIds (list) –

        A list of result IDs for the data quality results for the run.

        • (string) –

      • AdditionalDataSources (dict) –

        A map of reference strings to additional data sources you can specify for an evaluation run.

        • (string) –

          • (dict) –

            A data source (an Glue table) for which you want data quality results.

            • GlueTable (dict) –

              An Glue table.

              • DatabaseName (string) –

                A database name in the Glue Data Catalog.

              • TableName (string) –

                A table name in the Glue Data Catalog.

              • CatalogId (string) –

                A unique identifier for the Glue Data Catalog.

              • ConnectionName (string) –

                The name of the connection to the Glue Data Catalog.

              • AdditionalOptions (dict) –

                Additional options for the table. Currently there are two keys supported:

                • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

                • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

                • (string) –

                  • (string) –

            • DataQualityGlueTable (dict) –

              An Glue table for Data Quality Operations.

              • DatabaseName (string) –

                A database name in the Glue Data Catalog.

              • TableName (string) –

                A table name in the Glue Data Catalog.

              • CatalogId (string) –

                A unique identifier for the Glue Data Catalog.

              • ConnectionName (string) –

                The name of the connection to the Glue Data Catalog.

              • AdditionalOptions (dict) –

                Additional options for the table. Currently there are two keys supported:

                • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

                • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

                • (string) –

                  • (string) –

              • PreProcessingQuery (string) –

                SQL Query of SparkSQL format that can be used to pre-process the data for the table in Glue Data Catalog, before running the Data Quality Operation.

  • RunsNotFound (list) –

    A list of run IDs that were not found.

    • (string) –

Exceptions

• Glue.Client.exceptions.InvalidInputException

• Glue.Client.exceptions.OperationTimeoutException

• Glue.Client.exceptions.InternalServiceException