Tutorial: Get Started Using the Amazon A2I API - Amazon SageMaker AI
Create a Private Work TeamCreate a Human Review WorkflowCreate a Human Loop

Tutorial: Get Started Using the Amazon A2I API

This tutorial explains the API operations you can use to get started using Amazon A2I.

To use a Jupyter Notebook to run these operations, select a Jupyter Notebook from Use Cases and Examples Using Amazon A2I and use Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook to learn how to use it in a SageMaker AI notebook instance.

To learn more about the API operations you can use with Amazon A2I, see Use APIs in Amazon Augmented AI.

Create a Private Work Team

You can create a private work team and add yourself as a worker so that you can preview Amazon A2I.

If you are not familiar with Amazon Cognito, we recommend that you use the SageMaker AI console to create a private workforce and add yourself as a private worker. For instructions, see Step 1: Create a Work Team.

If you are familiar with Amazon Cognito, you can use the following instructions to create a private work team using the SageMaker API. After you create a work team, note the work team ARN (WorkteamArn).

To learn more about the private workforce and other available configurations, see Private workforce.

Create a private workforce

If you have not created a private workforce, you can do so using an Amazon Cognito user pool. Make sure that you have added yourself to this user pool. You can create a private work team using the AWS SDK for Python (Boto3) create_workforce function. For other language-specific SDKs, refer to the list in CreateWorkforce.

    
    response = client.create_workforce(
        CognitoConfig={
            "UserPool": "Pool_ID",
            "ClientId": "app-client-id"
        },
        WorkforceName="workforce-name"
    )
Create a private work team

After you have created a private workforce in the AWS Region to configure and start your human loop, you can create a private work team using the AWS SDK for Python (Boto3) create_workteam function. For other language-specific SDKs, refer to the list in CreateWorkteam.


    response = client.create_workteam(
        WorkteamName="work-team-name",
        WorkforceName= "workforce-name",
        MemberDefinitions=[
            {
                "CognitoMemberDefinition": {
                    "UserPool": "<aws-region>_ID",
                    "UserGroup": "user-group",
                    "ClientId": "app-client-id"
                },
            }
        ]
    )

Access your work team ARN as follows:


    workteamArn = response["WorkteamArn"]
List private work teams in your account

If you have already created a private work team, you can list all work teams in a given AWS Region in your account using the AWS SDK for Python (Boto3) list_workteams function. For other language-specific SDKs, refer to the list in ListWorkteams. 


    response = client.list_workteams()

If you have numerous work teams in your account, you may want to use MaxResults, SortBy, and NameContains to filter your results.

Create a Human Review Workflow

You can create a human review workflow using the Amazon A2I CreateFlowDefinition operation. Before you create your human review workflow, you need to create a human task UI. You can do this with the CreateHumanTaskUi operation.

If you are using Amazon A2I with the Amazon Textract or Amazon Rekognition integrations, you can specify activation conditions using a JSON.

Create a Human Task UI

If you are creating a human review workflow to be used with Amazon Textract or Amazon Rekognition integrations, you need to use and modify pre-made worker task template. For all custom integrations, you can use your own custom worker task template. Use the following table to learn how to create a human task UI using a worker task template for the two built-in integrations. Replace the template with your own to customize this request.

Amazon Textract – Key-value pair extraction

To learn more about this template, see Custom Template Example for Amazon Textract.

template = r"""
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
{% capture s3_uri %}http://s3.amazonaws.com/{{ task.input.aiServiceRequest.document.s3Object.bucket }}/{{ task.input.aiServiceRequest.document.s3Object.name }}{% endcapture %}
<crowd-form>
  <crowd-textract-analyze-document 
      src="{{ s3_uri | grant_read_access }}" 
      initial-value="{{ task.input.selectedAiServiceResponse.blocks }}" 
      header="Review the key-value pairs listed on the right and correct them if they don"t match the following document." 
      no-key-edit="" 
      no-geometry-edit="" 
      keys="{{ task.input.humanLoopContext.importantFormKeys }}" 
      block-types='["KEY_VALUE_SET"]'>
    <short-instructions header="Instructions">
        <p>Click on a key-value block to highlight the corresponding key-value pair in the document.
        </p><p><br></p>
        <p>If it is a valid key-value pair, review the content for the value. If the content is incorrect, correct it.
        </p><p><br></p>
        <p>The text of the value is incorrect, correct it.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/correct-value-text.png">
        </p><p><br></p>
        <p>A wrong value is identified, correct it.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/correct-value.png">
        </p><p><br></p>
        <p>If it is not a valid key-value relationship, choose No.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/not-a-key-value-pair.png">
        </p><p><br></p>
        <p>If you can’t find the key in the document, choose Key not found.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/key-is-not-found.png">
        </p><p><br></p>
        <p>If the content of a field is empty, choose Value is blank.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/value-is-blank.png">
        </p><p><br></p>
        <p><strong>Examples</strong></p>
        <p>Key and value are often displayed next or below to each other.
        </p><p><br></p>
        <p>Key and value displayed in one line.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/sample-key-value-pair-1.png">
        </p><p><br></p>
        <p>Key and value displayed in two lines.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/sample-key-value-pair-2.png">
        </p><p><br></p>
        <p>If the content of the value has multiple lines, enter all the text without line break. 
        Include all value text even if it extends beyond the highlight box.</p>
        <p><img src="https://assets.crowd.aws/images/a2i-console/multiple-lines.png"></p>
    </short-instructions>
    <full-instructions header="Instructions"></full-instructions>
  </crowd-textract-analyze-document>
</crowd-form>
"""
Amazon Rekognition – Image moderation

To learn more about this template, see Custom Template Example for Amazon Rekognition.

template = r"""
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
{% capture s3_uri %}http://s3.amazonaws.com/{{ task.input.aiServiceRequest.image.s3Object.bucket }}/{{ task.input.aiServiceRequest.image.s3Object.name }}{% endcapture %}

<crowd-form>
  <crowd-rekognition-detect-moderation-labels
    categories='[
      {% for label in task.input.selectedAiServiceResponse.moderationLabels %}
        {
          name: "{{ label.name }}",
          parentName: "{{ label.parentName }}",
        },
      {% endfor %}
    ]'
    src="{{ s3_uri | grant_read_access }}"
    header="Review the image and choose all applicable categories."
  >
    <short-instructions header="Instructions">
      <style>
        .instructions {
          white-space: pre-wrap;
        }
      </style>
      <p class="instructions">Review the image and choose all applicable categories.
If no categories apply, choose None.

<b>Nudity</b>
Visuals depicting nude male or female person or persons

<b>Partial Nudity</b>
Visuals depicting covered up nudity, for example using hands or pose

<b>Revealing Clothes</b>
Visuals depicting revealing clothes and poses

<b>Physical Violence</b>
Visuals depicting violent physical assault, such as kicking or punching

<b>Weapon Violence</b>
Visuals depicting violence using weapons like firearms or blades, such as shooting

<b>Weapons</b>
Visuals depicting weapons like firearms and blades
    </short-instructions>

    <full-instructions header="Instructions"></full-instructions>
  </crowd-rekognition-detect-moderation-labels>
</crowd-form>"""
Custom Integration

The following is an example template that can be used in a custom integration. This template is used in this notebook, demonstrating a custom integration with Amazon Comprehend.

template = r"""
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>

<crowd-form>
    <crowd-classifier
      name="sentiment"
      categories='["Positive", "Negative", "Neutral", "Mixed"]'
      initial-value="{{ task.input.initialValue }}"
      header="What sentiment does this text convey?"
    >
      <classification-target>
        {{ task.input.taskObject }}
      </classification-target>
      
      <full-instructions header="Sentiment Analysis Instructions">
        <p><strong>Positive</strong> sentiment include: joy, excitement, delight</p>
        <p><strong>Negative</strong> sentiment include: anger, sarcasm, anxiety</p>
        <p><strong>Neutral</strong>: neither positive or negative, such as stating a fact</p>
        <p><strong>Mixed</strong>: when the sentiment is mixed</p>
      </full-instructions>

      <short-instructions>
       Choose the primary sentiment that is expressed by the text. 
      </short-instructions>
    </crowd-classifier>
</crowd-form>
"""

Using the template specified above, you can create a template using the AWS SDK for Python (Boto3) create_human_task_ui function. For other language-specific SDKs, refer to the list in CreateHumanTaskUi. 

    
    response = client.create_human_task_ui(
        HumanTaskUiName="human-task-ui-name",
        UiTemplate={
            "Content": template
        }
    )

This response element contains the human task UI ARN. Save this as follows:


    humanTaskUiArn = response["HumanTaskUiArn"]

Create JSON to specify activation conditions

For Amazon Textract and Amazon Rekognition built-in integrations, you can save activation conditions in a JSON object and use this in your CreateFlowDefinition request.

Next, select a tab to see example activation conditions you can use for these built-in integrations. For additional information about activation condition options, see JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI.

Amazon Textract – Key-value pair extraction

This example specifies conditions for specific keys (such as Mail address) in the document. If Amazon Textract's confidence falls outside of the thresholds set here, the document is sent to a human for review, with the specific keys that initiated the human loop prompted to the worker.


      import json  

      humanLoopActivationConditions = json.dumps(
        {
            "Conditions": [
                {
                  "Or": [
                    
                    {
                        "ConditionType": "ImportantFormKeyConfidenceCheck",
                        "ConditionParameters": {
                            "ImportantFormKey": "Mail address",
                            "ImportantFormKeyAliases": ["Mail Address:","Mail address:", "Mailing Add:","Mailing Addresses"],
                            "KeyValueBlockConfidenceLessThan": 100,
                            "WordBlockConfidenceLessThan": 100
                        }
                    },
                    {
                        "ConditionType": "MissingImportantFormKey",
                        "ConditionParameters": {
                            "ImportantFormKey": "Mail address",
                            "ImportantFormKeyAliases": ["Mail Address:","Mail address:","Mailing Add:","Mailing Addresses"]
                        }
                    },
                    {
                        "ConditionType": "ImportantFormKeyConfidenceCheck",
                        "ConditionParameters": {
                            "ImportantFormKey": "Phone Number",
                            "ImportantFormKeyAliases": ["Phone number:", "Phone No.:", "Number:"],
                            "KeyValueBlockConfidenceLessThan": 100,
                            "WordBlockConfidenceLessThan": 100
                        }
                    },
                    {
                      "ConditionType": "ImportantFormKeyConfidenceCheck",
                      "ConditionParameters": {
                        "ImportantFormKey": "*",
                        "KeyValueBlockConfidenceLessThan": 100,
                        "WordBlockConfidenceLessThan": 100
                      }
                    },
                    {
                      "ConditionType": "ImportantFormKeyConfidenceCheck",
                      "ConditionParameters": {
                        "ImportantFormKey": "*",
                        "KeyValueBlockConfidenceGreaterThan": 0,
                        "WordBlockConfidenceGreaterThan": 0
                      }
                    }
            ]
        }
            ]
        }
    )
Amazon Rekognition – Image moderation

The human loop activation conditions used here are tailored towards Amazon Rekognition content moderation; they are based on the confidence thresholds for the Suggestive and Female Swimwear Or Underwear moderation labels.


        import json  

        humanLoopActivationConditions = json.dumps(
        {
            "Conditions": [
                {
                  "Or": [
                    {
                        "ConditionType": "ModerationLabelConfidenceCheck",
                        "ConditionParameters": {
                            "ModerationLabelName": "Suggestive",
                            "ConfidenceLessThan": 98
                        }
                    },
                    {
                        "ConditionType": "ModerationLabelConfidenceCheck",
                        "ConditionParameters": {
                            "ModerationLabelName": "Female Swimwear Or Underwear",
                            "ConfidenceGreaterThan": 98
                        }
                    }
                  ]
               }
            ]
        }
    )

Create a human review workflow

This section gives an example of the CreateFlowDefinition AWS SDK for Python (Boto3) request using the resources created in the previous sections. For other language-specific SDKs, refer to the list in CreateFlowDefinition. Use the tabs in the following table to see the requests to create a human review workflow for Amazon Textract and Amazon Rekognition built-in integrations.

Amazon Textract – Key-value pair extraction

If you use the built-in integration with Amazon Textract, you must specify "AWS/Textract/AnalyzeDocument/Forms/V1" for "AwsManagedHumanLoopRequestSource" in HumanLoopRequestSource. 


    response = client.create_flow_definition(
        FlowDefinitionName="human-review-workflow-name",
        HumanLoopRequestSource={
            "AwsManagedHumanLoopRequestSource": "AWS/Textract/AnalyzeDocument/Forms/V1"
        }, 
        HumanLoopActivationConfig={
            "HumanLoopActivationConditionsConfig": {
                "HumanLoopActivationConditions": humanLoopActivationConditions
            }
        },
        HumanLoopConfig={
            "WorkteamArn": workteamArn,
            "HumanTaskUiArn": humanTaskUiArn,
            "TaskTitle": "Document entry review",
            "TaskDescription": "Review the document and instructions. Complete the task",
            "TaskCount": 1,
            "TaskAvailabilityLifetimeInSeconds": 43200,
            "TaskTimeLimitInSeconds": 3600,
            "TaskKeywords": [
                "document review",
            ],
        },
        OutputConfig={
            "S3OutputPath": "s3://amzn-s3-demo-bucket/prefix/",
        },
        RoleArn="arn:aws:iam::<account-number>:role/<role-name>",
        Tags=[
            {
                "Key": "string",
                "Value": "string"
            },
        ]
    )
Amazon Rekognition – Image moderation

If you use the built-in integration with Amazon Rekognition, you must specify "AWS/Rekognition/DetectModerationLabels/Image/V3" for "AwsManagedHumanLoopRequestSource" in HumanLoopRequestSource.


    response = client.create_flow_definition(
        FlowDefinitionName="human-review-workflow-name",
        HumanLoopRequestSource={
            "AwsManagedHumanLoopRequestSource": "AWS/Rekognition/DetectModerationLabels/Image/V3"
        }, 
        HumanLoopActivationConfig={
            "HumanLoopActivationConditionsConfig": {
                "HumanLoopActivationConditions": humanLoopActivationConditions
            }
        },
        HumanLoopConfig={
            "WorkteamArn": workteamArn,
            "HumanTaskUiArn": humanTaskUiArn,
            "TaskTitle": "Image content moderation",
            "TaskDescription": "Review the image and instructions. Complete the task",
            "TaskCount": 1,
            "TaskAvailabilityLifetimeInSeconds": 43200,
            "TaskTimeLimitInSeconds": 3600,
            "TaskKeywords": [
                "content moderation",
            ],
        },
        OutputConfig={
            "S3OutputPath": "s3://amzn-s3-demo-bucket/prefix/",
        },
        RoleArn="arn:aws:iam::<account-number>:role/<role-name>",
        Tags=[
            {
                "Key": "string",
                "Value": "string"
            },
        ]
    )
Custom Integration

If you use a custom integration, exclude the following parameters: HumanLoopRequestSource, HumanLoopActivationConfig.


    response = client.create_flow_definition(
        FlowDefinitionName="human-review-workflow-name",
        HumanLoopConfig={
            "WorkteamArn": workteamArn,
            "HumanTaskUiArn": humanTaskUiArn,
            "TaskTitle": "Image content moderation",
            "TaskDescription": "Review the image and instructions. Complete the task",
            "TaskCount": 1,
            "TaskAvailabilityLifetimeInSeconds": 43200,
            "TaskTimeLimitInSeconds": 3600,
            "TaskKeywords": [
                "content moderation",
            ],
        },
        OutputConfig={
            "S3OutputPath": "s3://amzn-s3-demo-bucket/prefix/",
        },
        RoleArn="arn:aws:iam::<account-number>:role/<role-name>",
        Tags=[
            {
                "Key": "string",
                "Value": "string"
            },
        ]
    )

After you create a human review workflow, you can retrieve the flow definition ARN from the response:


    humanReviewWorkflowArn = response["FlowDefinitionArn"]

Create a Human Loop

The API operation you use to start a human loop depends on the Amazon A2I integration you use.

  • If you use the Amazon Textract built-in integration, you use the AnalyzeDocument operation.

  • If you use the Amazon Rekognition built-in integration, you use the DetectModerationLabels operation.

  • If you use a custom integration, you use the StartHumanLoop operation.

Select your task type in the following table to see example requests for Amazon Textract and Amazon Rekognition using the AWS SDK for Python (Boto3).

Amazon Textract – Key-value pair extraction

The following example uses the AWS SDK for Python (Boto3) to call analyze_document in us-west-2. Replace the italicized red text with your resources. Include the DataAttributes parameter if you are using the Amazon Mechanical Turk workforce. For more information, see the analyze_document documention in the AWS SDK for Python (Boto) API Reference.


   response = client.analyze_document(
         Document={"S3Object": {"Bucket": "amzn-s3-demo-bucket", "Name": "document-name.pdf"},
         HumanLoopConfig={
            "FlowDefinitionArn":"arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name",
            "HumanLoopName":"human-loop-name",
            "DataAttributes" : {ContentClassifiers:["FreeOfPersonallyIdentifiableInformation"|"FreeOfAdultContent"]}
         }
         FeatureTypes=["FORMS"]
    )

Human loops are only created if Amazon Textract's confidence for document analysis task meets the activation conditions you specified in your human review workflow. You can check the response element to determine if a human loop has been created. To see everything included in this response, see HumanLoopActivationOutput.


    if "HumanLoopArn" in analyzeDocumentResponse["HumanLoopActivationOutput"]:
        # A human loop has been started!
        print(f"A human loop has been started with ARN: {analyzeDocumentResponse["HumanLoopActivationOutput"]["HumanLoopArn"]}"
Amazon Rekognition – Image moderation

The following example uses the AWS SDK for Python (Boto3) to call detect_moderation_labels in us-west-2. Replace the italicized red text with your resources. Include the DataAttributes parameter if you are using the Amazon Mechanical Turk workforce. For more information, see the detect_moderation_labels documention in the AWS SDK for Python (Boto) API Reference.


   response = client.detect_moderation_labels(
            Image={"S3Object":{"Bucket": "amzn-s3-demo-bucket", "Name": "image-name.png"}},
            HumanLoopConfig={
               "FlowDefinitionArn":"arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name",
               "HumanLoopName":"human-loop-name",
               "DataAttributes":{ContentClassifiers:["FreeOfPersonallyIdentifiableInformation"|"FreeOfAdultContent"]}
             }
    )

Human loops are only created if Amazon Rekognition's confidence for an image moderation task meets the activation conditions you specified in your human review workflow. You can check the response element to determine if a human loop has been created. To see everything included in this response, see HumanLoopActivationOutput.


    if "HumanLoopArn" in response["HumanLoopActivationOutput"]:
        # A human loop has been started!
        print(f"A human loop has been started with ARN: {response["HumanLoopActivationOutput"]["HumanLoopArn"]}")
Custom Integration

The following example uses the AWS SDK for Python (Boto3) to call start_human_loop in us-west-2. Replace the italicized red text with your resources. Include the DataAttributes parameter if you are using the Amazon Mechanical Turk workforce. For more information, see the start_human_loop documention in the AWS SDK for Python (Boto) API Reference.


   response = client.start_human_loop(
        HumanLoopName= "human-loop-name",
        FlowDefinitionArn= "arn:aws:sagemaker:us-west-2:111122223333:flow-definition/flow-definition-name",
        HumanLoopInput={"InputContent": inputContentJson},
        DataAttributes={"ContentClassifiers":["FreeOfPersonallyIdentifiableInformation"|"FreeOfAdultContent"]}
   )

This example stores input content in the variable inputContentJson. Assume that the input content contains two elements: a text blurb and sentiment (such as Positive, Negative, or Neutral), and it is formatted as follows:


    inputContent = {
        "initialValue": sentiment,
         "taskObject": blurb
     }

The keys initialValue and taskObject must correspond to the keys used in the liquid elements of the worker task template. Refer to the custom template in Create a Human Task UI to see an example.

To create inputContentJson, do the following: 


    import json
    
    inputContentJson = json.dumps(inputContent)

A human loop starts each time you call start_human_loop. To check the status of your human loop, use describe_human_loop: 


    human_loop_info = a2i.describe_human_loop(HumanLoopName="human_loop_name")
    print(f"HumanLoop Status: {resp["HumanLoopStatus"]}")
    print(f"HumanLoop Output Destination: {resp["HumanLoopOutput"]}")