從 CSV 檔案建立分類資訊清單檔案

從 CSV 檔案建立清單檔案

1. 建立 CSV 檔案，每一資料列中包含以下欄位 (每個影像一個資料列)。請勿將標題資料列新增至 CSV 檔案。

   欄位 1	欄位 2
   影像名稱或 Amazon S3 路徑影像。例如：s3://s3bucket/circuitboard/train/anomaly/train-anomaly_10.jpg。您不能混合使用具有 Amazon S3 路徑的影像和不具有 Amazon S3 路徑的影像。	影像的異常分類 (normal 或 anomaly)。

   例如：s3://s3bucket/circuitboard/train/anomaly/image_10.jpg,anomaly 或 image_11.jpg,normal

2. 儲存 CSV 檔案。

3. 執行下列 Python 指令碼。提供下列引數：

   • csv_file — 您在步驟 1 中建立的 CSV 檔案。

   • (選用) --s3-path s3://path_to_folder/ — 要新增至影像檔案名稱的 Amazon S3 路徑 (欄位 1)。如果欄位 1 中的影像尚未包含 S3 路徑，請使用 --s3-path。

   # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
   # SPDX-License-Identifier:  Apache-2.0
   """
   Purpose
   Shows how to create an Amazon Lookout for Vision manifest file from a CSV file.
   The CSV file format is image location,anomaly classification (normal or anomaly)
   For example:
   s3://s3bucket/circuitboard/train/anomaly/train_11.jpg,anomaly
   s3://s3bucket/circuitboard/train/normal/train_1.jpg,normal
   
   If necessary, use the bucket argument to specify the Amazon S3 bucket folder for the images.
   """
   
   from datetime import datetime, timezone
   import argparse
   import logging
   import csv
   import os
   import json
   
   logger = logging.getLogger(__name__)
   
   
   def check_errors(csv_file):
       """
       Checks for duplicate images and incorrect classifications in a CSV file.
       If duplicate images or invalid anomaly assignments are found, an errors CSV file
       and deduplicated CSV file are created. Only the first
       occurrence of a duplicate is recorded. Other duplicates are recorded in the errors file.
       :param csv_file: The source CSV file
       :return: True if errors or duplicates are found, otherwise false.
       """
   
       logger.info("Checking %s.", csv_file)
   
       errors_found = False
       errors_file = f"{os.path.splitext(csv_file)[0]}_errors.csv"
       deduplicated_file = f"{os.path.splitext(csv_file)[0]}_deduplicated.csv"
   
       with open(csv_file, 'r', encoding="UTF-8") as input_file,\
               open(deduplicated_file, 'w', encoding="UTF-8") as dedup,\
               open(errors_file, 'w', encoding="UTF-8") as errors:
   
           reader = csv.reader(input_file, delimiter=',')
           dedup_writer = csv.writer(dedup)
           error_writer = csv.writer(errors)
           line = 1
           entries = set()
           for row in reader:
   
               # Skip empty lines.
               if not ''.join(row).strip():
                   continue
   
               # Record any incorrect classifications.
               if not row[1].lower() == "normal" and not row[1].lower() == "anomaly":
                   error_writer.writerow(
                       [line, row[0], row[1], "INVALID_CLASSIFICATION"])
                   errors_found = True
   
               # Write first image entry to dedup file and record duplicates.
               key = row[0]
               if key not in entries:
                   dedup_writer.writerow(row)
                   entries.add(key)
               else:
                   error_writer.writerow([line, row[0], row[1], "DUPLICATE"])
                   errors_found = True
               line += 1
   
       if errors_found:
           logger.info("Errors found check %s.", errors_file)
       else:
           os.remove(errors_file)
           os.remove(deduplicated_file)
   
       return errors_found
   
   
   def create_manifest_file(csv_file, manifest_file, s3_path):
       """
       Read a CSV file and create an Amazon Lookout for Vision classification manifest file.
       :param csv_file: The source CSV file.
       :param manifest_file: The name of the manifest file to create.
       :param s3_path: The Amazon S3 path to the folder that contains the images.
       """
       logger.info("Processing CSV file %s.", csv_file)
   
       image_count = 0
       anomalous_count = 0
   
       with open(csv_file, newline='', encoding="UTF-8") as csvfile,\
           open(manifest_file, "w", encoding="UTF-8") as output_file:
   
           image_classifications = csv.reader(
               csvfile, delimiter=',', quotechar='|')
   
           # Process each row (image) in the CSV file.
           for row in image_classifications:
               # Skip empty lines.
               if not ''.join(row).strip():
                   continue
   
               source_ref = str(s3_path) + row[0]
               classification = 0
   
               if row[1].lower() == 'anomaly':
                   classification = 1
                   anomalous_count += 1
   
              # Create the JSON line.
               json_line = {}
               json_line['source-ref'] = source_ref
               json_line['anomaly-label'] = str(classification)
   
               metadata = {}
               metadata['confidence'] = 1
               metadata['job-name'] = "labeling-job/anomaly-classification"
               metadata['class-name'] = row[1]
               metadata['human-annotated'] = "yes"
               metadata['creation-date'] = datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%S.%f')
               metadata['type'] = "groundtruth/image-classification"
   
               json_line['anomaly-label-metadata'] = metadata
   
               output_file.write(json.dumps(json_line))
               output_file.write('\n')
               image_count += 1
   
       logger.info("Finished creating manifest file %s.\n"
                   "Images: %s\nAnomalous: %s",
                   manifest_file,
                   image_count,
                   anomalous_count)
       return image_count, anomalous_count
   
   
   def add_arguments(parser):
       """
       Add command line arguments to the parser.
       :param parser: The command line parser.
       """
   
       parser.add_argument(
           "csv_file", help="The CSV file that you want to process."
       )
   
       parser.add_argument(
           "--s3_path", help="The Amazon S3 bucket and folder path for the images."
           " If not supplied, column 1 is assumed to include the Amazon S3 path.", required=False
       )
   
   
   def main():
   
       logging.basicConfig(level=logging.INFO,
                           format="%(levelname)s: %(message)s")
   
       try:
   
           # Get command line arguments.
           parser = argparse.ArgumentParser(usage=argparse.SUPPRESS)
           add_arguments(parser)
           args = parser.parse_args()
           s3_path = args.s3_path
           if s3_path is None:
               s3_path = ""
   
           csv_file = args.csv_file
           csv_file_no_extension = os.path.splitext(csv_file)[0]
           manifest_file = csv_file_no_extension + '.manifest'
   
           # Create manifest file if there are no duplicate images.
           if check_errors(csv_file):
               print(f"Issues found. Use {csv_file_no_extension}_errors.csv "\
                   "to view duplicates and errors.")
               print(f"{csv_file}_deduplicated.csv contains the first"\
                   "occurrence of a duplicate.\n"
                     "Update as necessary with the correct information.")
               print(f"Re-run the script with {csv_file_no_extension}_deduplicated.csv")
           else:
               print('No duplicates found. Creating manifest file.')
   
               image_count, anomalous_count = create_manifest_file(csv_file, manifest_file, s3_path)
   
               print(f"Finished creating manifest file: {manifest_file} \n")
   
               normal_count = image_count-anomalous_count
               print(f"Images processed: {image_count}")
               print(f"Normal: {normal_count}")
               print(f"Anomalous: {anomalous_count}")
   
       except FileNotFoundError as err:
           logger.exception("File not found.:%s", err)
           print(f"File not found: {err}. Check your input CSV file.")
   
   if __name__ == "__main__":
       main()
   
   

4. 如果發生重複的影像或發生分類錯誤：

   1. 使用錯誤檔案來更新重複資料刪除的 CSV 檔案或輸入 CSV 檔案。

   2. 使用更新的重複資料刪除 CSV 檔案或更新的輸入 CSV 檔案再次執行指令碼。

5. 如果您打算使用測試資料集，請重複步驟 1–4 來為您的測試資料集建立資訊清單檔案。

6. 如有必要，請將映像從您的電腦複製到您在 CSV 檔案 （或--s3-path命令列中指定的） 第 1 欄指定的 Amazon S3 儲存貯體路徑。若要複製映像，請在命令提示中輸入下列命令。

   aws s3 cp --recursive your-local-folder/ s3://your-target-S3-location/

7. 遵循 的指示來使用資訊清單檔案建立資料集 （主控台）建立資料集。如果您使用 AWS SDK，請參閱 使用資訊清單檔案 (SDK) 建立資料集。