FileUniqueness
File Uniqueness allows you to ensure that there are no duplicate files in the data you have received from your data producers.
It gathers the following data statistics
-
Total number of files in the folder
-
The Uniqueness Ratio of the files
Dataset.*.FileUniquenessRatio: 1.00, Dataset.*.FileCount: 8.00
Find duplicate files in a folder:
FileUniqueness "s3://bucket/" > 0.5 FileUniqueness "s3://bucket/folder/" = 1
Inferring folder names directly from data frames to detect duplicates:
You don't always have to provide a file path. For instance, when you are authoring the rule in the AWS Glue Data Catalog, it may be hard to find which folders the catalog tables are using. AWS Glue Data Quality can find the specific folders or files used to populate your data frame.
FileUniqueness > 0.5 FileUniqueness with threshold = 1
There are a few considerations:
-
In AWS Glue ETL, you must have the EvaluateDataQuality Transform immediately after an Amazon S3 or AWS Glue Data Catalog transform.
-
This rule will not work in AWS Glue Interactive Sessions.
