使用清单文件导入图像 - Rekognition

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

使用清单文件导入图像

您可以使用 Amazon G SageMaker round Truth 格式的清单文件创建数据集。你可以使用 Amazon G SageMaker round Truth 任务中的清单文件。如果您的图像和标签不是 G SageMaker round Truth 清单文件的格式,则可以创建 SageMaker 格式清单文件并使用它来导入已贴标签的图像。

CreateDataset操作已更新,允许您在创建新数据集时选择性地指定标签。标签是键值对,可用于对资源进行分类和管理。

使用 G SageMaker round Truth 清单文件创建数据集(控制台)

以下过程向您展示如何使用 G SageMaker round Truth 格式的清单文件创建数据集。

  1. 通过执行下列操作之一,为训练数据集创建清单文件:

    如果要创建测试数据集,请重复步骤 1 以创建测试数据集。

  2. 打开亚马逊 Rekognition 控制台,网址为https://console.aws.amazon.com/rekognition/

  3. 选择使用自定义标签

  4. 选择开始

  5. 在左侧导航窗格中,选择项目

  6. 项目页面上,选择要向其添加数据集的项目。此时将显示项目的详细信息页面。

  7. 选择创建数据集。此时将显示创建数据集页面。

  8. 开始配置中,选择从单个数据集开始从训练数据集开始。要创建更高质量的模型,建议从单独的训练和测试数据集开始。

    Single dataset
    1. 训练数据集详细信息部分,选择导入 SageMaker由 Ground Truth 标注的图像

    2. .manifest 文件位置中,输入您在步骤 1 中创建的清单文件的位置。

    3. 选择创建数据集。这时会打开项目的数据集页面。

    Separate training and test datasets
    1. 训练数据集详细信息部分,选择导入 SageMaker由 Ground Truth 标注的图像

    2. .manifest 文件位置中,输入您在步骤 1 中创建的训练数据集清单文件的位置。

    3. 测试数据集详细信息部分,选择导入 SageMaker 由 Ground Truth 标注的图像

      注意

      训练数据集和测试数据集可以有不同的图像源。

    4. .manifest 文件位置中,输入您在步骤 1 中创建的测试数据集清单文件的位置。

    5. 选择创建数据集。这时会打开项目的数据集页面。

  9. 如果需要添加或更改标签,请执行标注图像中的操作。

  10. 按照训练模型(控制台)中的步骤训练您的模型。

使用 G SageMaker round Truth 清单文件创建数据集 (SDK)

以下过程向您展示如何使用清单文件创建训练或测试数据集CreateDatasetAPI。

您可以使用现有的清单文件,例如 G SageMaker round Truth 任务的输出,也可以创建自己的清单文件

  1. 如果您尚未这样做,请安装并配置 AWS CLI 和 AWS SDKs。有关更多信息,请参阅 步骤 4:设置 AWS CLI 以及 AWS SDKs

  2. 通过执行下列操作之一,为训练数据集创建清单文件:

    如果要创建测试数据集,请重复步骤 2 以创建测试数据集。

  3. 使用以下示例代码创建训练和测试数据集。

    AWS CLI

    使用以下代码创建数据集。替换以下内容:

    • project_arn— 您要向其添加测试数据集的项目的。ARN

    • type-要创建的数据集的类型(TRAIN或TEST)

    • bucket:包含数据集清单文件的存储桶。

    • manifest_file:清单文件的路径和文件名。

    aws rekognition create-dataset --project-arn project_arn \ --dataset-type type \ --dataset-source '{ "GroundTruthManifest": { "S3Object": { "Bucket": "bucket", "Name": "manifest_file" } } }' \ --profile custom-labels-access --tags '{"key1": "value1", "key2": "value2"}'
    Python

    使用以下值创建数据集。提供以下命令行参数:

    • project_arn— 您要向其添加测试数据集的项目的。ARN

    • dataset_type:要创建的数据集的类型(traintest)。

    • bucket:包含数据集清单文件的存储桶。

    • manifest_file:清单文件的路径和文件名。

    #Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. #PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-custom-labels-developer-guide/blob/master/LICENSE-SAMPLECODE.) import argparse import logging import time import json import boto3 from botocore.exceptions import ClientError logger = logging.getLogger(__name__) def create_dataset(rek_client, project_arn, dataset_type, bucket, manifest_file): """ Creates an Amazon Rekognition Custom Labels dataset. :param rek_client: The Amazon Rekognition Custom Labels Boto3 client. :param project_arn: The ARN of the project in which you want to create a dataset. :param dataset_type: The type of the dataset that you want to create (train or test). :param bucket: The S3 bucket that contains the manifest file. :param manifest_file: The path and filename of the manifest file. """ try: #Create the project logger.info("Creating %s dataset for project %s",dataset_type, project_arn) dataset_type = dataset_type.upper() dataset_source = json.loads( '{ "GroundTruthManifest": { "S3Object": { "Bucket": "' + bucket + '", "Name": "' + manifest_file + '" } } }' ) response = rek_client.create_dataset( ProjectArn=project_arn, DatasetType=dataset_type, DatasetSource=dataset_source ) dataset_arn=response['DatasetArn'] logger.info("dataset ARN: %s",dataset_arn) finished=False while finished is False: dataset=rek_client.describe_dataset(DatasetArn=dataset_arn) status=dataset['DatasetDescription']['Status'] if status == "CREATE_IN_PROGRESS": logger.info("Creating dataset: %s ",dataset_arn) time.sleep(5) continue if status == "CREATE_COMPLETE": logger.info("Dataset created: %s", dataset_arn) finished=True continue if status == "CREATE_FAILED": error_message = f"Dataset creation failed: {status} : {dataset_arn}" logger.exception(error_message) raise Exception (error_message) error_message = f"Failed. Unexpected state for dataset creation: {status} : {dataset_arn}" logger.exception(error_message) raise Exception(error_message) return dataset_arn except ClientError as err: logger.exception("Couldn't create dataset: %s",err.response['Error']['Message']) raise def add_arguments(parser): """ Adds command line arguments to the parser. :param parser: The command line parser. """ parser.add_argument( "project_arn", help="The ARN of the project in which you want to create the dataset." ) parser.add_argument( "dataset_type", help="The type of the dataset that you want to create (train or test)." ) parser.add_argument( "bucket", help="The S3 bucket that contains the manifest file." ) parser.add_argument( "manifest_file", help="The path and filename of the manifest file." ) def main(): logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") try: #Get command line arguments. parser = argparse.ArgumentParser(usage=argparse.SUPPRESS) add_arguments(parser) args = parser.parse_args() print(f"Creating {args.dataset_type} dataset for project {args.project_arn}") #Create the dataset. session = boto3.Session(profile_name='custom-labels-access') rekognition_client = session.client("rekognition") dataset_arn=create_dataset(rekognition_client, args.project_arn, args.dataset_type, args.bucket, args.manifest_file) print(f"Finished creating dataset: {dataset_arn}") except ClientError as err: logger.exception("Problem creating dataset: %s", err) print(f"Problem creating dataset: {err}") if __name__ == "__main__": main()
    Java V2

    使用以下值创建数据集。提供以下命令行参数:

    • project_arn— 您要向其添加测试数据集的项目的。ARN

    • dataset_type:要创建的数据集的类型(traintest)。

    • bucket:包含数据集清单文件的存储桶。

    • manifest_file:清单文件的路径和文件名。

    /* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: Apache-2.0 */ package com.example.rekognition; import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.rekognition.RekognitionClient; import software.amazon.awssdk.services.rekognition.model.CreateDatasetRequest; import software.amazon.awssdk.services.rekognition.model.CreateDatasetResponse; import software.amazon.awssdk.services.rekognition.model.DatasetDescription; import software.amazon.awssdk.services.rekognition.model.DatasetSource; import software.amazon.awssdk.services.rekognition.model.DatasetStatus; import software.amazon.awssdk.services.rekognition.model.DatasetType; import software.amazon.awssdk.services.rekognition.model.DescribeDatasetRequest; import software.amazon.awssdk.services.rekognition.model.DescribeDatasetResponse; import software.amazon.awssdk.services.rekognition.model.GroundTruthManifest; import software.amazon.awssdk.services.rekognition.model.RekognitionException; import software.amazon.awssdk.services.rekognition.model.S3Object; import java.util.logging.Level; import java.util.logging.Logger; public class CreateDatasetManifestFiles { public static final Logger logger = Logger.getLogger(CreateDatasetManifestFiles.class.getName()); public static String createMyDataset(RekognitionClient rekClient, String projectArn, String datasetType, String bucket, String name) throws Exception, RekognitionException { try { logger.log(Level.INFO, "Creating {0} dataset for project : {1} from s3://{2}/{3} ", new Object[] { datasetType, projectArn, bucket, name }); DatasetType requestDatasetType = null; switch (datasetType) { case "train": requestDatasetType = DatasetType.TRAIN; break; case "test": requestDatasetType = DatasetType.TEST; break; default: logger.log(Level.SEVERE, "Could not create dataset. Unrecognized dataset type: {0}", datasetType); throw new Exception("Could not create dataset. Unrecognized dataset type: " + datasetType); } GroundTruthManifest groundTruthManifest = GroundTruthManifest.builder() .s3Object(S3Object.builder().bucket(bucket).name(name).build()).build(); DatasetSource datasetSource = DatasetSource.builder().groundTruthManifest(groundTruthManifest).build(); CreateDatasetRequest createDatasetRequest = CreateDatasetRequest.builder().projectArn(projectArn) .datasetType(requestDatasetType).datasetSource(datasetSource).build(); CreateDatasetResponse response = rekClient.createDataset(createDatasetRequest); boolean created = false; do { DescribeDatasetRequest describeDatasetRequest = DescribeDatasetRequest.builder() .datasetArn(response.datasetArn()).build(); DescribeDatasetResponse describeDatasetResponse = rekClient.describeDataset(describeDatasetRequest); DatasetDescription datasetDescription = describeDatasetResponse.datasetDescription(); DatasetStatus status = datasetDescription.status(); logger.log(Level.INFO, "Creating dataset ARN: {0} ", response.datasetArn()); switch (status) { case CREATE_COMPLETE: logger.log(Level.INFO, "Dataset created"); created = true; break; case CREATE_IN_PROGRESS: Thread.sleep(5000); break; case CREATE_FAILED: String error = "Dataset creation failed: " + datasetDescription.statusAsString() + " " + datasetDescription.statusMessage() + " " + response.datasetArn(); logger.log(Level.SEVERE, error); throw new Exception(error); default: String unexpectedError = "Unexpected creation state: " + datasetDescription.statusAsString() + " " + datasetDescription.statusMessage() + " " + response.datasetArn(); logger.log(Level.SEVERE, unexpectedError); throw new Exception(unexpectedError); } } while (created == false); return response.datasetArn(); } catch (RekognitionException e) { logger.log(Level.SEVERE, "Could not create dataset: {0}", e.getMessage()); throw e; } } public static void main(String[] args) { String datasetType = null; String bucket = null; String name = null; String projectArn = null; String datasetArn = null; final String USAGE = "\n" + "Usage: " + "<project_arn> <dataset_type> <dataset_arn>\n\n" + "Where:\n" + " project_arn - the ARN of the project that you want to add copy the datast to.\n\n" + " dataset_type - the type of the dataset that you want to create (train or test).\n\n" + " bucket - the S3 bucket that contains the manifest file.\n\n" + " name - the location and name of the manifest file within the bucket.\n\n"; if (args.length != 4) { System.out.println(USAGE); System.exit(1); } projectArn = args[0]; datasetType = args[1]; bucket = args[2]; name = args[3]; try { // Get the Rekognition client RekognitionClient rekClient = RekognitionClient.builder() .credentialsProvider(ProfileCredentialsProvider.create("custom-labels-access")) .region(Region.US_WEST_2) .build(); // Create the dataset datasetArn = createMyDataset(rekClient, projectArn, datasetType, bucket, name); System.out.println(String.format("Created dataset: %s", datasetArn)); rekClient.close(); } catch (RekognitionException rekError) { logger.log(Level.SEVERE, "Rekognition client error: {0}", rekError.getMessage()); System.exit(1); } catch (Exception rekError) { logger.log(Level.SEVERE, "Error: {0}", rekError.getMessage()); System.exit(1); } } }
  4. 如果需要添加或更改标签,请参阅管理标签 (SDK)

  5. 按照训练模型(SDK)中的步骤训练您的模型。

创建数据集请求

以下是 CreateDataset 操作请求的格式:

{ "DatasetSource": { "DatasetArn": "string", "GroundTruthManifest": { "S3Object": { "Bucket": "string", "Name": "string", "Version": "string" } } }, "DatasetType": "string", "ProjectArn": "string", "Tags": { "string": "string" } }