Segment each instance of each class of objects in the image.
GOOD EXAMPLES
Good because A, B, C.
BAD EXAMPLES
Bad because X, Y, Z.
- Read the task carefully and inspect the image.
- Read the options and review the examples provided to understand more about the labels.
- Choose the appropriate label that best suits the image.
Good example
Enter description to explain the correct label to the workers
Bad example
Enter description of an incorrect label
- Read the task carefully and inspect the image.
- Read the options and review the examples provided to understand more about the labels.
- Choose the appropriate labels that best suit the image.
Good example
Enter description to explain the correct label to the workers
Bad example
Enter description of an incorrect label
Add header
Add Instructions
", "fullInstruction":"Add additional instructions.
"} ``` The following is an example of an [AWS Python SDK (Boto3) request](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_labeling_job) to create a labeling job in the US East (N. Virginia) Region. All parameters in red should be replaced with your specifications and resources. ``` response = client.create_labeling_job( LabelingJobName='example-ner-labeling-job', LabelAttributeName='label', InputConfig={ 'DataSource': { 'S3DataSource': { 'ManifestS3Uri': 's3://bucket/path/manifest-with-input-data.json' } }, 'DataAttributes': { 'ContentClassifiers': [ 'FreeOfPersonallyIdentifiableInformation'|'FreeOfAdultContent', ] } }, OutputConfig={ 'S3OutputPath': 's3://bucket/path/file-to-store-output-data', 'KmsKeyId': 'string' }, RoleArn='arn:aws:iam::*:role/*', LabelCategoryConfigS3Uri='s3://bucket/path/label-categories.json', StoppingConditions={ 'MaxHumanLabeledObjectCount': 123, 'MaxPercentageOfInputDatasetLabeled': 123 }, HumanTaskConfig={ 'WorkteamArn': 'arn:aws:sagemaker:region:*:workteam/private-crowd/*', 'UiConfig': { 'HumanTaskUiArn': 'arn:aws:sagemaker:us-east-1:394669845002:human-task-ui/NamedEntityRecognition' }, 'PreHumanTaskLambdaArn': 'arn:aws:lambda:us-east-1:432418664414:function:PRE-NamedEntityRecognition', 'TaskKeywords': [ 'Named entity Recognition', ], 'TaskTitle': 'Named entity Recognition task', 'TaskDescription': 'Apply the labels provided to specific words or phrases within the larger text block.', 'NumberOfHumanWorkersPerDataObject': 1, 'TaskTimeLimitInSeconds': 28800, 'TaskAvailabilityLifetimeInSeconds': 864000, 'MaxConcurrentTaskCount': 1000, 'AnnotationConsolidationConfig': { 'AnnotationConsolidationLambdaArn': 'arn:aws:lambda:us-east-1:432418664414:function:ACS-NamedEntityRecognition' }, Tags=[ { 'Key': 'string', 'Value': 'string' }, ] ) ``` ### Provide Worker Instructions in a Label Category Configuration File You must provide worker instructions in the label category configuration file you identify with the `LabelCategoryConfigS3Uri` parameter in `CreateLabelingJob`. You can use these instructions to provide details about the task you want workers to perform and help them use the tool efficiently. You provide short and long instructions using `shortInstruction` and `fullInstruction` in the `instructions` parameter, respectively. To learn more about these instruction types, see [Create instruction pages](sms-creating-instruction-pages.md). The following is an example of a label category configuration file with instructions that can be used for a named entity recognition labeling job. ``` { "document-version": "2018-11-28", "labels": [ { "label": "label1", "shortDisplayName": "L1" }, { "label": "label2", "shortDisplayName": "L2" }, { "label": "label3", "shortDisplayName": "L3" }, { "label": "label4", "shortDisplayName": "L4" }, { "label": "label5", "shortDisplayName": "L5" } ], "instructions": { "shortInstruction": "Enter description of the labels that workers have to choose from
Add examples to help workers understand the label
", "fullInstruction": "- Read the text carefully.
- Highlight words, phrases, or sections of the text.
- Choose the label that best matches what you have highlighted.
- To change a label, choose highlighted text and select a new label.
- To remove a label from highlighted text, choose the X next to the abbreviated label name on the highlighted text.
- You can select all of a previously highlighted text, but not a portion of it.
- Read the text carefully.
- Read the examples to understand more about the options.
- Choose the appropriate labels that best suit the text.
Enter description of the labels that workers have to choose from
Add examples to help workers understand the label
- Read the text carefully.
- Read the examples to understand more about the options.
- Choose the appropriate labels that best suit the text.
Enter description of the labels that workers have to choose from
Add examples to help workers understand the label
- Read the task carefully and inspect the video.
- Read the options and review the examples provided to understand more about the labels.
- Choose the appropriate label that best suits the video.
Good example
Enter description to explain the correct label to the workers
Bad example
Enter description of an incorrect label
Use this area to add more detailed worker instructions.
" }, "annotationType": [ { "type": "BoundingBox" }, { "type": "Cuboid" } ] } ``` **Note** You need to provide `BoundingBox` and `Cuboid` as annotationType in the label category configuration file to create a 3D-2D object tracking job. # View the worker task interface for a 3D-2D object tracking labeling job Ground Truth provides workers with a web portal and tools to complete your 3D-2D object tracking annotation tasks. When you create the labeling job, you provide the Amazon Resource Name (ARN) for a pre-built Ground Truth UI in the `HumanTaskUiArn` parameter. To use the UI when you create a labeling job for this task type using the API, you need to provide the `HumanTaskUiArn`. You can preview and interact with the worker UI when you create a labeling job through the API. The annotating tools are a part of the worker task interface. They are not available for the preview interface. The following image demonstrates the worker task interface used for the 3D-2D point cloud object tracking annotation task. ![\[The worker task interface used for the 3D-2D point cloud object tracking annotation task.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/sms-sensor-fusion.png) When interpolation is enabled by default. After a worker adds a single cuboid, that cuboid is replicated in all frames of the sequence with the same ID. If the worker adjusts the cuboid in another frame, Ground Truth interpolates the movement of that object and adjust all cuboids between the manually adjusted frames. Additionally, using the camera view section, a cuboid can be shown with a projection (using to B button for "toggle labels" in the camera view) that provides the worker with a reference from the camera images. The accuracy of the cuboid to image projection is based on accuracy of calibrations captured in the extrinsic and intrinsinc data. If you provide camera data for sensor fusion, images are matched up with scenes in point cloud frames. Note that the camera data should be time synchronized with the point cloud data to ensure an accurate depiction of point cloud to imagery over each frame in the sequence as shown in the following image. ![\[The manifest file, the worker portal with point cloud data and the camera data.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/3d_2d_link_ss.png) The manifest file holds the extrinsic and intrinsic data and the pose to allow the cuboid projection on the camera image to be shown by using the **P button**. Worker can navigate in the 3D scene using their keyboard and mouse. They can: + Double click on specific objects in the point cloud to zoom into them. + Use a mouse-scroller or trackpad to zoom in and out of the point cloud. + Use both keyboard arrow keys and Q, E, A, and D keys to move Up, Down, Left, Right. Use keyboard keys W and S to zoom in and out. Once a worker places a cuboids in the 3D scene, a side-view appears with the three projected side views: top, side, and front. These side-views show points in and around the placed cuboid and help workers refine cuboid boundaries in that area. Workers can zoom in and out of each of those side-views using their mouse. The worker should first select the cuboid to draw a corresponding bounding box on any of the camera views. This links the cuboid and the bounding box with a common name and unique ID. The worker can also first draw a bounding box, select it and draw the corresponding cuboid to link them. Additional view options and features are available. See the [worker instruction page](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-worker-instructions-object-tracking.html) for a comprehensive overview of the Worker UI. ## Worker tools Workers can navigate through the 3D point cloud by zooming in and out, and moving in all directions around the cloud using the mouse and keyboard shortcuts. If workers click on a point in the point cloud, the UI automatically zooms into that area. Workers can use various tools to draw 3D cuboid around objects. For more information, see **Assistive Labeling Tools** in the following discussion. After workers have placed a 3D cuboid in the point cloud, they can adjust these cuboids to fit tightly around cars using a variety of views: directly in the 3D point cloud, in a side-view featuring three zoomed-in perspectives of the point cloud around the box, and if you include images for sensor fusion, directly in the 2D image. Additional view options enable workers to easily hide or view label text, a ground mesh, and additional point attributes. Workers can also choose between perspective and orthogonal projections. **Assistive Labeling Tools** Ground Truth helps workers annotate 3D point clouds faster and more accurately using UX, machine learning and computer vision powered assistive labeling tools for 3D point cloud object tracking tasks. The following assistive labeling tools are available for this task type: + **Label autofill** – When a worker adds a cuboid to a frame, a cuboid with the same dimensions, orientation and xyz position is automatically added to all frames in the sequence. + **Label interpolation** – After a worker has labeled a single object in two frames, Ground Truth uses those annotations to interpolate the movement of that object between all the frames. Label interpolation can be turned on and off. It is on by default. For example, if a worker working with 5 frames adds a cuboid in frame 2, it is copied to all the 5 frames. If the worker then makes adjustments in frame 4, frame 2 and 4 now act as two points, through which a line is fit. The cuboid is then interpolated in frames 1,3 and 5. + **Bulk label and attribute management** – Workers can add, delete, and rename annotations, label category attributes, and frame attributes in bulk. + Workers can manually delete annotations for a given object before and after a frame, or in all frames. For example, a worker can delete all labels for an object after frame 10 if that object is no longer located in the scene after that frame. + If a worker accidentally bulk deletes all annotations for a object, they can add them back. For example, if a worker deletes all annotations for an object before frame 100, they can bulk add them to those frames. + Workers can rename a label in one frame and all 3D cuboids assigned that label are updated with the new name across all frames. + Workers can use bulk editing to add or edit label category attributes and frame attributes in multiple frames. + **Snapping** – Workers can add a cuboid around an object and use a keyboard shortcut or menu option to have Ground Truth's autofit tool snap the cuboid tightly around the object's boundaries. + **Fit to ground** – After a worker adds a cuboid to the 3D scene, the worker can automatically snap the cuboid to the ground. For example, the worker can use this feature to snap a cuboid to the road or sidewalk in the scene. + **Multi-view labeling** – After a worker adds a 3D cuboid to the 3D scene, a side-panel displays front and two side perspectives to help the worker adjust the cuboid tightly around the object. Workers can annotation the 3D point cloud, the side panel and the adjustments appear in the other views in real time. + **Sensor fusion** – If you provide data for sensor fusion, workers can adjust annotations in the 3D scenes and in 2D images, and the annotations are projected into the other view in real time. To learn more about the data for sensor fusion, see [Understand Coordinate Systems and Sensor Fusion](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-sensor-fusion-details.html#sms-point-cloud-sensor-fusion). + **Auto-merge cuboids **– Workers can automatically merge two cuboids across all frames if they determine that cuboids with different labels actually represent a single object. + **View options **– Enables workers to easily hide or view label text, a ground mesh, and additional point attributes like color or intensity. Workers can also choose between perspective and orthogonal projections. # Output data for a 3D-2D object tracking labeling job When you create a 3D-2D object tracking labeling job, tasks are sent to workers. When these workers complete their tasks, their annotations are written to the Amazon S3 bucket you specified when you created the labeling job. The output data format determines what you see in your Amazon S3 bucket when your labeling job status ([LabelingJobStatus](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeLabelingJob.html#API_DescribeLabelingJob_ResponseSyntax)) is `Completed`. If you are a new user of Ground Truth, see [Labeling job output data](sms-data-output.md) to learn more about the Ground Truth output data format. To learn about the 3D-2D point cloud object tracking output data format, see [3D-2D object tracking point cloud object tracking output](sms-data-output.md#sms-output-3d-2d-point-cloud-object-tracking). # 3D point cloud labeling jobs overview This topic provides an overview of the unique features of a Ground Truth 3D point cloud labeling job. You can use the 3D point cloud labeling jobs to have workers label objects in a 3D point cloud generated from a 3D sensors like LiDAR and depth cameras or generated from 3D reconstruction by stitching images captured by an agent like a drone. ## Job pre-processing time When you create a 3D point cloud labeling job, you need to provide an [input manifest file](sms-point-cloud-input-data.md). The input manifest file can be: + A *frame input manifest file* that has a single point cloud frame on each line. + A *sequence input manifest file* that has a single sequence on each line. A sequence is defined as a temporal series of point cloud frames. For both types of manifest files, *job pre-processing time* (that is, the time before Ground Truth starts sending tasks to your workers) depends on the total number and size of point cloud frames you provide in your input manifest file. For frame input manifest files, this is the number of lines in your manifest file. For sequence manifest files, this is the number of frames in each sequence multiplied by the total number of sequences, or lines, in your manifest file. Additionally, the number of points per point cloud and the number of fused sensor data objects (like images) factor into job pre-processing times. On average, Ground Truth can pre-process 200 point cloud frames in approximately 5 minutes. If you create a 3D point cloud labeling job with a large number of point cloud frames, you might experience longer job pre-processing times. For example, if you create a sequence input manifest file with 4 point cloud sequences, and each sequence contains 200 point clouds, Ground Truth pre-processes 800 point clouds and so your job pre-processing time might be around 20 minutes. During this time, your labeling job status is `InProgress`. While your 3D point cloud labeling job is pre-processing, you receive CloudWatch messages notifying you of the status of your job. To identify these messages, search for `3D_POINT_CLOUD_PROCESSING_STATUS` in your labeling job logs. For **frame input manifest files**, your CloudWatch logs will have a message similar to the following: ``` { "labeling-job-name": "example-point-cloud-labeling-job", "event-name": "3D_POINT_CLOUD_PROCESSING_STATUS", "event-log-message": "datasetObjectId from: 0 to 10, status: IN_PROGRESS" } ``` The event log message, `datasetObjectId from: 0 to 10, status: IN_PROGRESS` identifies the number of frames from your input manifest that have been processed. You receive a new message every time a frame has been processed. For example, after a single frame has processed, you receive another message that says `datasetObjectId from: 1 to 10, status: IN_PROGRESS`. For **sequence input manifest files**, your CloudWatch logs will have a message similar to the following: ``` { "labeling-job-name": "example-point-cloud-labeling-job", "event-name": "3D_POINT_CLOUD_PROCESSING_STATUS", "event-log-message": "datasetObjectId: 0, status: IN_PROGRESS" } ``` The event log message, `datasetObjectId from: 0, status: IN_PROGRESS` identifies the number of sequences from your input manifest that have been processed. You receive a new message every time a sequence has been processed. For example, after a single sequence has processed, you receive a message that says `datasetObjectId from: 1, status: IN_PROGRESS` as the next sequence begins processing. ## Job completion times 3D point cloud labeling jobs can take workers hours to complete. You can set the total amount of time that workers can work on each task when you create a labeling job. The maximum time you can set for workers to work on tasks is 7 days. The default value is 3 days. It is strongly recommended that you create tasks that workers can complete within 12 hours. Workers must keep the worker UI open while working on a task. They can save work as they go and Ground Truth will save their work every 15 minutes. When using the SageMaker AI `CreateLabelingJob` API operation, set the total time a task is available to workers in the `TaskTimeLimitInSeconds` parameter of `HumanTaskConfig`. When you create a labeling job in the console, you can specify this time limit when you select your workforce type and your work team. ## Workforces When you create a 3D point cloud labeling job, you need to specify a work team that will complete your point cloud annotation tasks. You can choose a work team from a private workforce of your own workers, or from a vendor workforce that you select in the AWS Marketplace. You cannot use the Amazon Mechanical Turk workforce for 3D point cloud labeling jobs. To learn more about vendor workforce, see [Subscribe to vendor workforces](sms-workforce-management-vendor.md). To learn how to create and manage a private workforce, see [Private workforce](sms-workforce-private.md). ## Worker user interface (UI) Ground Truth provides a worker user interface (UI), tools, and assistive labeling features to help workers complete your 3D point cloud labeling tasks. You can preview the worker UI when you create a labeling job in the console. When you create a labeling job using the API operation `CreateLabelingJob`, you must provide an ARN provided by Ground Truth in the parameter [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UiConfig.html#sagemaker-Type-UiConfig-UiTemplateS3Uri](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UiConfig.html#sagemaker-Type-UiConfig-UiTemplateS3Uri) to specify the worker UI for your task type. You can use `HumanTaskUiArn` with the SageMaker AI [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_RenderUiTemplate.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_RenderUiTemplate.html) API operation to preview the worker UI. You provide worker instructions, labels, and optionally, label category attributes that are displayed in the worker UI. ### Label category attributes When you create a 3D point cloud object tracking or object detection labeling job, you can add one or more *label category attributes*. You can add *frame attributes* to all 3D point cloud task types: + **Label category attribute** – A list of options (strings), a free form text box, or a numeric field associated with one or more labels. It is used by workers to to provide metadata about a label. + **Frame attribute** – A list of options (strings), a free form text box, or a numeric field that appears on each point cloud frame a worker is sent to annotate. It is used by workers to provide metadata about frames. Additionally, you can use label and frame attributes to have workers verify labels in a 3D point cloud label verification job. Use the following sections to learn more about these attributes. To learn how to add label category and frame attributes to a labeling job, use the **Create Labeling Job** section on the [task type page](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-task-types) of your choice. #### Label category attributes Add label category attributes to labels to give workers the ability to provide more information about the annotations they create. A label category attribute is added to an individual label, or to all labels. When a label category attribute is applied to all labels it is referred to as a *global label category attribute*. For example, if you add the label category *car*, you might also want to capture additional data about your labeled cars, such as if they are occluded or the size of the car. You can capture this metadata using label category attributes. In this example, if you added the attribute *occluded* to the car label category, you can assign *partial*, *completely*, *no* to the *occluded* attribute and enable workers to select one of these options. When you create a label verification job, you add labels category attributes to each label you want workers to verify. #### Frame attributes Add frame attributes to give workers the ability to provide more information about individual point cloud frames. You can specify up to 10 frame attributes, and these attributes will appear on all frames. For example, you can add a frame attribute that allows workers to enter a number. You may want to use this attribute to have workers identify the number of objects they see in a particular frame. In another example, you may want to provide a free-form text box to give workers the ability to provide a free form answer to a question. When you create a label verification job, you can add one or more frame attributes to ask workers to provide feedback on all labels in a point cloud frame. ### Worker instructions You can provide worker instructions to help your workers complete your point cloud labeling tasks. You might want to use these instructions to do the following: + Best practices and things to avoid when annotating objects. + Explanation of the label category attributes provided (for object detection and object tracking tasks), and how to use them. + Advice on how to save time while labeling by using keyboard shortcuts. You can add your worker instructions using the SageMaker AI console while creating a labeling job. If you create a labeling job using the API operation `CreateLabelingJob`, you specify worker instructions in your label category configuration file. In addition to your instructions, Ground Truth provides a link to help workers navigate and use the worker portal. View these instructions by selecting the task type on [Worker instructions](sms-point-cloud-worker-instructions.md). ### Declining tasks Workers are able to decline tasks. Workers decline a task if the instructions are not clear, input data is not displaying correctly, or if they encounter some other issue with the task. If the number of workers per dataset object ([https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_HumanTaskConfig.html#sagemaker-Type-HumanTaskConfig-NumberOfHumanWorkersPerDataObject](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_HumanTaskConfig.html#sagemaker-Type-HumanTaskConfig-NumberOfHumanWorkersPerDataObject)) decline the task, the data object is marked as expired and will not be sent to additional workers. # 3D point cloud labeling job permission requirements When you create a 3D point cloud labeling job, in addition to the permission requirements found in [Assign IAM Permissions to Use Ground Truth](sms-security-permission.md), you must add a CORS policy to your S3 bucket that contains your input manifest file. ## Add a CORS permission policy to S3 bucket When you create a 3D point cloud labeling job, you specify buckets in S3 where your input data and manifest file are located and where your output data will be stored. These buckets may be the same. You must attach the following Cross-origin resource sharing (CORS) policy to your input and output buckets. If you use the Amazon S3 console to add the policy to your bucket, you must use the JSON format. **JSON** ``` [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "HEAD", "PUT" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [ "Access-Control-Allow-Origin" ], "MaxAgeSeconds": 3000 } ] ``` **XML** ```