View a markdown version of this page

HIT Review Policy - Amazon Mechanical Turk
DescriptionHIT Review Policy ElementsParameter ElementsMapEntry ElementsExamples

Amazon Mechanical Turk will permanently close on September 30, 2026. For Workers and Requesters currently using the service, visit our Amazon Mechanical Turk help page to learn how you can prepare for this closure.

HIT Review Policy

Description

HIT Review Policy data structures represent HIT review policies, which you specify when you create a Human Intelligence Task (HIT). For more information about Review Policies, see Review Policies.

The following two types of HIT Review Policy data structures are used when calling the CreateHIT operation.

  • AssignmentReviewPolicy data structures set the review results and actions at the Assignment level. For more information, see Assignment Review Policies.

  • HITReviewPolicy data structures set the review results and actions at the HIT-level. For more information, see HIT Review Policies.

The HIT Review Policy data structure is used in the following operation:

  • CreateHIT

HIT Review Policy Elements

The HIT Review Policy data structures can contain the following elements.

Name Description Required

PolicyName

Name of a Review Policy. For policy names and descriptions, see Assignment Review Policies and HIT Review Policies.

Type: String

Constraints: SimplePlurality/2011-09-01 or ScoreMyKnownAnswers/2011-09-01

Yes

Parameter

Name of the parameter from the Review policy.

Type: Parameter data structure, for a description see the next section Parameter Elements.

Parameter Elements

The Parameter data structure contains the elements described in the following table.

Name Description Required

Key

Name of the parameter from the list of Review Polices. For a list of valid parameter names, see Assignment Review Policies and HIT Review Policies.

Type: String

Constraints: none

Default: none

Value

Value of the parameter.

Type: Varies.

MapEntry

This data structure is the data type for the AnswerKey parameter of the ScoreMyKnownAnswers/2011-09-01 Review Policy, see Assignment Review Policies.

Type: MapEntry data structure, for a description, see the next section MapEntry Elements.

MapEntry Elements

The MapEntry element contains the elements described in the following table.

Name Description Required

Key

The QuestionID from the HIT that is used to identify which question requires Mechanical Turk to score as part of the ScoreMyKnownAnswers/2011-09-01 Review Policy. Can be up to 255 bytes in length.

Type: String none

Value

The answer to the question specified in the MapEntry Key element.

Note: You can provide multiple Value elements, but the Worker must match all values in order for the answer to be scored correctly.

Type: String

Examples

The following example request shows the structure of a CreateHIT request using HIT Review Policy data structures. The example request is followed by an example response.

Sample Request

The following is an example CreateHIT request.


{	HITTypeId:"T100CN9P324W00EXAMPLE",
    Question:URL-encoded question data,
    LifetimeInSeconds:604800,
    AssignmentReviewPolicy:{
        PolicyName:"ScoreMyKnownAnswers/2011-09-01",
        Parameters:[
			{ Key:"AnswerKey",
		  		MapEntries:[
					{Key: "QuestionId1", Values:["B"]},
					{Key: "QuestionId2", Value:["A"]},
					{Key: "QuestionId3", Value:["C"]},
					{Key: "QuestionId4", Value:["A"]}
					]
			}, 
			{ Key: "ApproveIfKnownAnswerScoreIsAtLeast",
			  Values:["79"]
			}
			{ Key: "ExtendIfKnownAnswerScoreIsLessThan",
			  Values:["79"]
			}			
			{ Key: "ExtendMaximumAssignments",
			  Values:["3"]
			}			
		]
	},
    HITReviewPolicy:{
        PolicyName:"SimplePlurality/2011-09-01",
        Parameters:[
			{
				Key:"QuestionIDs",
				// Add upto 15 questions
				Values:["questionid1", "questionid2", "questionid3", "questionid4", "questionid5"]
			},
			{
				Key:"QuestionAgreementThreshold",
				Values:["100"]
			}
		]
	}
}