從專案匯出資料集 (SDK)

從專案匯出資料集 (SDK)

1. 如果您尚未這麼做，請安裝並設定 AWS CLI 和 AWS SDKs。如需詳細資訊，請參閱步驟 4：設定 AWS CLI 和 SDK AWS SDKs。

2. 判斷資料集匯出的目的地 Amazon S3 路徑。請確定目的地位於 Amazon Lookout for Vision 支援的AWS 區域中。若要建立新的 Amazon S3 儲存貯體，請參閱建立儲存貯體。

3. 確定使用者具有資料集匯出目的地 Amazon S3 路徑的存取許可，以及來源專案資料集中影像檔案的 S3 位置。您可以使用下列政策，假設影像檔案可以位於任何位置。將儲存貯體/路徑取代為資料集匯出的目的地儲存貯體和路徑。

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "PutExports",
               "Effect": "Allow",
               "Action": [
                   "S3:PutObjectTagging",
                   "S3:PutObject"
               ],
               "Resource": "arn:aws:s3:::bucket/path/*"
           },
           {
               "Sid": "GetSourceRefs",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:GetObjectTagging",
                   "s3:GetObjectVersion"
               ],
               "Resource": "*"
           }
       ]
   }

   若要提供存取權，請新增權限至您的使用者、群組或角色：

   • 中的使用者和群組 AWS IAM Identity Center：

     建立權限合集。請按照 AWS IAM Identity Center 使用者指南 中的 建立權限合集 說明進行操作。

   • 透過身分提供者在 IAM 中管理的使用者：

     建立聯合身分的角色。遵循「IAM 使用者指南」的為第三方身分提供者 (聯合) 建立角色中的指示。

   • IAM 使用者：

     • 建立您的使用者可擔任的角色。請按照「IAM 使用者指南」的為 IAM 使用者建立角色中的指示。

     • (不建議) 將政策直接附加至使用者，或將使用者新增至使用者群組。請遵循 IAM 使用者指南的新增許可到使用者 (主控台) 中的指示。

4. 將以下程式碼儲存到名為 dataset_export.py 的檔案。

   
   """
   Purpose
   
   Shows how to export the datasets (manifest files and images)
   from an Amazon Lookout for Vision project to a new Amazon 
   S3 location.
   """
   
   import argparse
   import json
   import logging
   
   import boto3
   from botocore.exceptions import ClientError
   
   logger = logging.getLogger(__name__)
   
   
   def copy_file(s3_resource, source_file, destination_file):
       """
       Copies a file from a source Amazon S3 folder to a destination
       Amazon S3 folder.
       The destination can be in a different S3 bucket.
       :param s3: An Amazon S3 Boto3 resource.
       :param source_file: The Amazon S3 path to the source file.
       :param destination_file: The destination Amazon S3 path for
       the copy operation.
       """
   
       source_bucket, source_key = source_file.replace("s3://", "").split("/", 1)
       destination_bucket, destination_key = destination_file.replace("s3://", "").split(
           "/", 1
       )
   
       try:
           bucket = s3_resource.Bucket(destination_bucket)
           dest_object = bucket.Object(destination_key)
           dest_object.copy_from(CopySource={"Bucket": source_bucket, "Key": source_key})
           dest_object.wait_until_exists()
           logger.info("Copied %s to %s", source_file, destination_file)
       except ClientError as error:
           if error.response["Error"]["Code"] == "404":
               error_message = (
                   f"Failed to copy {source_file} to "
                   f"{destination_file}. : {error.response['Error']['Message']}"
               )
               logger.warning(error_message)
               error.response["Error"]["Message"] = error_message
           raise
   
   
   def upload_manifest_file(s3_resource, manifest_file, destination):
       """
       Uploads a manifest file to a destination Amazon S3 folder.
       :param s3: An Amazon S3 Boto3 resource.
       :param manifest_file: The manifest file that you want to upload.
       :destination: The Amazon S3 folder location to upload the manifest
       file to.
       """
   
       destination_bucket, destination_key = destination.replace("s3://", "").split("/", 1)
   
       bucket = s3_resource.Bucket(destination_bucket)
   
       put_data = open(manifest_file, "rb")
       obj = bucket.Object(destination_key + manifest_file)
   
       try:
           obj.put(Body=put_data)
           obj.wait_until_exists()
           logger.info("Put manifest file '%s' to bucket '%s'.", obj.key, obj.bucket_name)
       except ClientError:
           logger.exception(
               "Couldn't put manifest file '%s' to bucket '%s'.", obj.key, obj.bucket_name
           )
           raise
       finally:
           if getattr(put_data, "close", None):
               put_data.close()
   
   
   def get_dataset_types(lookoutvision_client, project):
       """
       Determines the types of the datasets (train or test) in an
       Amazon Lookout for Vision project.
       :param lookoutvision_client: A Lookout for Vision Boto3 client.
       :param project: The Lookout for Vision project that you want to check.
       :return: The dataset types in the project.
       """
   
       try:
           response = lookoutvision_client.describe_project(ProjectName=project)
   
           datasets = []
   
           for dataset in response["ProjectDescription"]["Datasets"]:
               if dataset["Status"] in ("CREATE_COMPLETE", "UPDATE_COMPLETE"):
                   datasets.append(dataset["DatasetType"])
           return datasets
   
       except lookoutvision_client.exceptions.ResourceNotFoundException:
           logger.exception("Project %s not found.", project)
           raise
   
   
   def process_json_line(s3_resource, entry, dataset_type, destination):
       """
       Creates a JSON line for a new manifest file, copies image and mask to
       destination.
       :param s3_resource: An Amazon S3 Boto3 resource.
       :param entry: A JSON line from the manifest file.
       :param dataset_type: The type (train or test) of the dataset that
       you want to create the manifest file for.
       :param destination: The destination Amazon S3 folder for the manifest
       file and dataset images.
       :return: A JSON line with details for the destination location.
       """
       entry_json = json.loads(entry)
   
       print(f"source: {entry_json['source-ref']}")
   
       # Use existing folder paths to ensure console added image names don't clash.
       bucket, key = entry_json["source-ref"].replace("s3://", "").split("/", 1)
       logger.info("Source location: %s/%s", bucket, key)
   
       destination_image_location = destination + dataset_type + "/images/" + key
   
       copy_file(s3_resource, entry_json["source-ref"], destination_image_location)
   
       # Update JSON for writing.
       entry_json["source-ref"] = destination_image_location
   
       if "anomaly-mask-ref" in entry_json:
           source_anomaly_ref = entry_json["anomaly-mask-ref"]
           mask_bucket, mask_key = source_anomaly_ref.replace("s3://", "").split("/", 1)
   
           destination_mask_location = destination + dataset_type + "/masks/" + mask_key
           entry_json["anomaly-mask-ref"] = destination_mask_location
   
           copy_file(s3_resource, source_anomaly_ref, entry_json["anomaly-mask-ref"])
   
       return entry_json
   
   
   def write_manifest_file(
       lookoutvision_client, s3_resource, project, dataset_type, destination
   ):
       """
       Creates a manifest file for a dataset. Copies the manifest file and
       dataset images (and masks, if present) to the specified Amazon S3 destination.
       :param lookoutvision_client: A Lookout for Vision Boto3 client.
       :param project: The Lookout for Vision project that you want to use.
       :param dataset_type: The type (train or test) of the dataset that
       you want to create the manifest file for.
       :param destination: The destination Amazon S3 folder for the manifest file
       and dataset images.
       """
   
       try:
           # Create a reusable Paginator
           paginator = lookoutvision_client.get_paginator("list_dataset_entries")
   
           # Create a PageIterator from the Paginator
           page_iterator = paginator.paginate(
               ProjectName=project,
               DatasetType=dataset_type,
               PaginationConfig={"PageSize": 100},
           )
   
           output_manifest_file = dataset_type + ".manifest"
   
           # Create manifest file then upload to Amazon S3 with images.
           with open(output_manifest_file, "w", encoding="utf-8") as manifest_file:
               for page in page_iterator:
                   for entry in page["DatasetEntries"]:
                       try:
                           entry_json = process_json_line(
                               s3_resource, entry, dataset_type, destination
                           )
   
                           manifest_file.write(json.dumps(entry_json) + "\n")
   
                       except ClientError as error:
                           if error.response["Error"]["Code"] == "404":
                               print(error.response["Error"]["Message"])
                               print(f"Excluded JSON line: {entry}")
                           else:
                               raise
           upload_manifest_file(
               s3_resource, output_manifest_file, destination + "datasets/"
           )
   
       except ClientError:
           logger.exception("Problem getting dataset_entries")
           raise
   
   
   def export_datasets(lookoutvision_client, s3_resource, project, destination):
       """
       Exports the datasets from an Amazon Lookout for Vision project to a specified
       Amazon S3 destination.
       :param project: The Lookout for Vision project that you want to use.
       :param destination: The destination Amazon S3 folder for the exported datasets.
       """
       # Add trailing backslash, if missing.
       destination = destination if destination[-1] == "/" else destination + "/"
   
       print(f"Exporting project {project} datasets to {destination}.")
   
       # Get each dataset and export to destination.
   
       dataset_types = get_dataset_types(lookoutvision_client, project)
       for dataset in dataset_types:
           logger.info("Copying %s dataset to %s.", dataset, destination)
   
           write_manifest_file(
               lookoutvision_client, s3_resource, project, dataset, destination
           )
   
       print("Exported dataset locations")
       for dataset in dataset_types:
           print(f"   {dataset}: {destination}datasets/{dataset}.manifest")
   
       print("Done.")
   
   
   def add_arguments(parser):
       """
       Adds command line arguments to the parser.
       :param parser: The command line parser.
       """
   
       parser.add_argument("project", help="The project that contains the dataset.")
       parser.add_argument("destination", help="The destination Amazon S3 folder.")
   
   
   def main():
       """
       Exports the datasets from an Amazon Lookout for Vision project to a
       destination Amazon S3 location.
       """
       logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
       parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
       add_arguments(parser)
   
       args = parser.parse_args()
   
       try:
           session = boto3.Session(profile_name="lookoutvision-access")
           lookoutvision_client = session.client("lookoutvision")
           s3_resource = session.resource("s3")
   
           export_datasets(
               lookoutvision_client, s3_resource, args.project, args.destination
           )
       except ClientError as err:
           logger.exception(err)
           print(f"Failed: {format(err)}")
   
   
   if __name__ == "__main__":
       main()
   
   

5. 執行程式碼。提供下列命令列參數：

   • 專案 – 包含您要匯出之資料集的來源專案名稱。

   • destination – 資料集的目的地 Amazon S3 路徑。

   例如 python dataset_export.py myproject s3://bucket/path/

6. 請注意程式碼顯示的清單檔案位置。您在步驟 8 中需要它們。

7. 遵循 中的指示，使用匯出的資料集建立新的 Lookout for Vision 專案建立您的專案。

8. 執行以下任意一項：

   • 依照 的指示，使用 Lookout for Vision 主控台為您的新專案建立資料集使用資訊清單檔案建立資料集 （主控台）。您不需要執行步驟 1–6。

     針對步驟 12，執行下列動作：

     1. 如果來源專案具有測試資料集，請選擇分開訓練和測試資料摘要，否則請選擇單一資料集。

     2. 對於 .manifest 檔案位置，輸入您在步驟 6 中記下的適當資訊清單檔案 （訓練或測試） 的位置。

   • 使用 CreateDataset 操作，使用 的程式碼為新專案建立資料集使用資訊清單檔案 (SDK) 建立資料集。針對 manifest_file 參數，請使用您在步驟 6 中記下的資訊清單檔案位置。如果來源專案具有測試資料集，請再次使用程式碼來建立測試資料集。

9. 如果您已準備好，請依照 的指示來訓練模型培訓您的模型。