# Integrating optimized services with Step Functions Your workflow can call optimized services directly using the `Resource` field of a `Task` state. The following topics explain the supported APIs, parameters, and request/response syntax in Amazon States Language for coordinating AWS services. Depending on workflow type and availability, your workflows call services using one of three service integration patterns: + [Request a Response (default)](connect-to-resource.md#connect-default) - wait for HTTP response, then go to the next state + [Run a Job (`.sync`)](connect-to-resource.md#connect-sync) - wait for the job to complete + [Wait for Callback (`.waitForTaskToken`)](connect-to-resource.md#connect-wait-token) - pause a workflow until a task token is returned Standard Workflows and Express Workflows support the same **integrations** but not the same **integration patterns**. + **Standard Workflows** support *Request Response* integrations. Certain services support *Run a Job (.sync)*, or *Wait for Callback (.waitForTaskToken)* , and both in some cases. See the following optimized integrations table for details. + **Express Workflows** only support *Request Response* integrations. To help decide between the two types, see [Choosing workflow type in Step Functions](choosing-workflow-type.md). **AWS SDK integrations in Step Functions** | Integrated service | Request Response | Run a Job - *.sync* | Wait for Callback - *.waitForTaskToken* | | --- | --- | --- | --- | | [Over two hundred services](supported-services-awssdk.md#supported-services-awssdk-list) | Standard & Express | Not supported | Standard | **Optimized integrations in Step Functions** | Integrated service | Request Response | Run a Job - *.sync* | Wait for Callback - *.waitForTaskToken* | | --- | --- | --- | --- | | [Amazon API Gateway](connect-api-gateway.md) | Standard & Express | Not supported | Standard | | [Amazon Athena](connect-athena.md) | Standard & Express | Standard | Not supported | | [AWS Batch](connect-batch.md) | Standard & Express | Standard | Not supported | | [Amazon Bedrock](connect-bedrock.md) | Standard & Express | Standard | Standard | | [AWS CodeBuild](connect-codebuild.md) | Standard & Express | Standard | Not supported | | [Amazon DynamoDB](connect-ddb.md) | Standard & Express | Not supported | Not supported | | [Amazon ECS/Fargate](connect-ecs.md) | Standard & Express | Standard | Standard | | [Amazon EKS](connect-eks.md) | Standard & Express | Standard | Standard | | [Amazon EMR](connect-emr.md) | Standard & Express | Standard | Not supported | | [Amazon EMR on EKS](connect-emr-eks.md) | Standard & Express | Standard | Not supported | | [Amazon EMR Serverless](connect-emr-serverless.md) | Standard & Express | Standard | Not supported | | [Amazon EventBridge](connect-eventbridge.md) | Standard & Express | Not supported | Standard | | [AWS Glue](connect-glue.md) | Standard & Express | Standard | Not supported | | [AWS Glue DataBrew](connect-databrew.md) | Standard & Express | Standard | Not supported | | [AWS Lambda](connect-lambda.md) | Standard & Express | Not supported | Standard | | [AWS Elemental MediaConvert](connect-mediaconvert.md) | Standard & Express | Standard | Not supported | | [Amazon SageMaker AI](connect-sagemaker.md) | Standard & Express | Standard | Not supported | | [Amazon SNS](connect-sns.md) | Standard & Express | Not supported | Standard | | [Amazon SQS](connect-sqs.md) | Standard & Express | Not supported | Standard | | [AWS Step Functions](connect-stepfunctions.md) | Standard & Express | Standard | Standard | # Create API Gateway REST APIs with Step Functions Learn how to use Amazon API Gateway to create, publish, maintain, and monitor HTTP and REST APIs with Step Functions. To integrate with API Gateway, you define a `Task` state in Step Functions that directly calls an API Gateway HTTP or API Gateway REST endpoint, without writing code or relying on other infrastructure. A `Task` state definition includes all the necessary information for the API call. You can also select different authorization methods. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized API Gateway integration** `apigateway:invoke:` has no equivalent in the AWS SDK service integration. Instead, the Optimized API Gateway service calls your API Gateway endpoint directly. ## API Gateway feature support The Step Functions API Gateway integration supports some, but not all API Gateway features. For a more detailed list of supported features, see the following. + Supported by both the Step Functions API Gateway REST API and API Gateway HTTP API integrations: + **Authorizers**: IAM (using [Signature Version 4](https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html)), No Auth, Lambda Authorizers (request-parameter based and token-based with custom header) + **API types**: Regional + **API management**: API Gateway API domain names, API stage, Path, Query Parameters, Request Body + Supported by the Step Functions API Gateway HTTP API integration. The Step Functions API Gateway REST API integration that provides the option for Edge-optimized APIs are not supported. + Unsupported by the Step Functions API Gateway integration: + **Authorizers**: Amazon Cognito, Native Open ID Connect / OAuth 2.0, Authorization header for token-based Lambda authorizers + **API types**: Private + **API management**: Custom domain names For more information about API Gateway and its HTTP and REST APIs, see the following. + The [Amazon API Gateway concepts](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-basic-concept.html) page. + [Choosing between HTTP APIs and REST APIs](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vs-rest.html) in the API Gateway developer guide. ## Request format When you create your `Task` state definition, Step Functions validates the parameters, builds the necessary URL to perform the call, then calls the API. The response includes the HTTP status code, headers and response body. The request format has both required and optional parameters. ### Required request parameters + `ApiEndpoint` + Type: `String` + The hostname of an API Gateway URL. The format is `.execute-api.region.amazonaws.com`. The API ID can only contain a combination of the following alphanumeric characters: `0123456789abcdefghijklmnopqrstuvwxyz` + `Method` + Type: `Enum` + The HTTP method, which must be one of the following: + `GET` + `POST` + `PUT` + `DELETE` + `PATCH` + `HEAD` + `OPTIONS` ### Optional request parameters + `Headers` + Type: `JSON` + HTTP headers allow a list of values associated with the same key. + `Stage` + Type: `String` + The name of the stage where the API is deployed to in API Gateway. It's optional for any HTTP API that uses the `$default` stage. + `Path` + Type: `String` + Path parameters that are appended after the API endpoint. + `QueryParameters` + Type: `JSON` + Query strings only allow a list of values associated with the same key. + `RequestBody` + Type: `JSON` or `String` + The HTTP Request body. Its type can be either a `JSON` object or `String`. `RequestBody` is only supported for `PATCH`, `POST`, and `PUT` HTTP methods. + `AllowNullValues` + Type: `BOOLEAN` – default value: `false` + With the default setting, any **null** values in the request input state will **not** be sent to your API. In the following example, the `category` field will **not** be included in the request, unless `AllowNullValues` is set to `true` in your state machine definition. ``` { "NewPet": { "type": "turtle", "price": 123, "category": null } } ``` **Note** By default, fields with **null** values in the request input state will **not** be sent to your API. You can force null values to be sent to your API by setting `AllowNullValues` to `true` in your state machine definition. + `AuthType` + Type: `JSON` + The authentication method. The default method is `NO_AUTH`. The allowed values are: + `NO_AUTH` + `IAM_ROLE` + `RESOURCE_POLICY` See **Authentication and authorization** for more information. **Note** For security considerations, the following HTTP header keys are not currently permitted: Anything prefixed with `X-Forwarded`, `X-Amz` or `X-Amzn`. `Authorization` `Connection` `Content-md5` `Expect` `Host` `Max-Forwards` `Proxy-Authenticate` `Server` `TE` `Transfer-Encoding` `Trailer` `Upgrade` `Via` `Www-Authenticate` The following code example shows how to invoke API Gateway using Step Functions. ``` { "Type": "Task", "Resource":"arn:aws:states:::apigateway:invoke", "Arguments": { "ApiEndpoint": "example.execute-api.us-east-1.amazonaws.com", "Method": "GET", "Headers": { "key": ["value1", "value2"] }, "Stage": "prod", "Path": "bills", "QueryParameters": { "billId": ["123456"] }, "RequestBody": {}, "AuthType": "NO_AUTH" } } ``` ## Authentication and authorization You can use the following authentication methods: + **No authorization**: Call the API directly with no authorization method. + **IAM role**: With this method, Step Functions assumes the role of the state machine, signs the request with [Signature Version 4](https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html) (SigV4), then calls the API. + **Resource policy**: Step Functions authenticates the request, and then calls the API. You must attach a resource policy to the API which specifies the following: 1. The state machine that will invoke API Gateway. **Important** You must specify your state machine to limit access to it. If you do not, then any state machine that authenticates its API Gateway request with **Resource policy** authentication to your API will be granted access. 1. That Step Functions is the service calling API Gateway: `"Service": "states.amazonaws.com"`. 1. The resource you want to access, including: + The *region*. + The *account-id* in the specified region. + The *api-id*. + The *stage-name*. + The *HTTP-VERB* (method). + The *resource-path-specifier*. For an example resource policy, see [IAM policies for Step Functions and API Gateway](#api-gateway-iam). For more information on the resource format, see [Resource format of permissions for executing API in API Gateway ](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-control-access-using-iam-policies-to-invoke-api.html#api-gateway-iam-policy-resource-format-for-executing-api) in the API Gateway Developer Guide. **Note** Resource policies are only supported for the REST API. ## Service integration patterns The API Gateway integration supports two service integration patterns: + [Request Response](connect-to-resource.md#connect-default), which is the default integration pattern. It lets Step Functions progress to the next step immediately after it receives an HTTP response. + [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) (`.waitForTaskToken`), which waits until a task token is returned with a payload. To use the `.waitForTaskToken` pattern, append .waitForTaskToken to the end of the **Resource** field of your task definition as shown in the following example: ``` { "Type": "Task", "Resource":"arn:aws:states:::apigateway:invoke.waitForTaskToken", "Arguments": { "ApiEndpoint": "example.execute-api.us-east-1.amazonaws.com", "Method": "POST", "Headers": { "TaskToken": "{% $states.context.Task.Token %}" }, "Stage": "prod", "Path": "bills/add", "QueryParameters": {}, "RequestBody": { "billId": "my-new-bill" }, "AuthType": "IAM_ROLE" } } ``` ## Output format The following output parameters are provided: | Name | Type | Description | | --- | --- | --- | | ResponseBody | JSON or String | The response body of the API call. | | Headers | JSON | The response headers. | | StatusCode | Integer | The HTTP status code of the response. | | StatusText | String | The status text of the response. | An example response: ``` { "ResponseBody": { "myBills": [] }, "Headers": { "key": ["value1", "value2"] }, "StatusCode": 200, "StatusText": "OK" } ``` ## Error handling When an error occurs, an `error` and `cause` is returned as follows: + If the HTTP status code is available, then the error will be returned in the format `ApiGateway.`. + If the HTTP status code is not available, then the error will be returned in the format `ApiGateway.`. In both cases, the `cause` is returned as a string. The following example shows a response where an error has occurred: ``` { "error": "ApiGateway.403", "cause": "{\"message\":\"Missing Authentication Token\"}" } ``` **Note** A status code of `2XX` indicates success, and no error will be returned. All other status codes or thrown exceptions will result in an error. ## IAM policies for calls to Amazon API Gateway The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). *Resources*: **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "execute-api:Invoke" ], "Resource": [ "arn:aws:execute-api:us-east-1:123456789012:ENDPOINT/STAGE/GET/pets", "arn:aws:execute-api:us-east-1:123456789012:ENDPOINT/STAGE/POST/pets" ], "Effect": "Allow" } ] } ``` The following code example shows a resource policy for calling API Gateway. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "states.amazonaws.com" }, "Action": "execute-api:Invoke", "Resource": "arn:aws:execute-api:us-east-1:123456789012:myApi-id/stage-name/HTTP-VERB/resource-path-specifier", "Condition": { "StringEquals": { "aws:SourceArn": [ "" ] } } } ] } ``` # Run Athena queries with Step Functions You can integrate AWS Step Functions with Amazon Athena to start and stop query execution and get query results with Step Functions. Using Step Functions, you can run ad-hoc or scheduled data queries, and retrieve results targeting your S3 data lakes. Athena is serverless, so there is no infrastructure to set up or manage, and you pay only for the queries you run. This page lists the supported Athena APIs and provides an example `Task` state to start an Athena query. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized Athena integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. There are no specific optimizations for the [Request Response](connect-to-resource.md#connect-default) integration pattern. The [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration pattern is not supported. To integrate AWS Step Functions with Amazon Athena, you use the provided Athena service integration APIs. The service integration APIs are the same as the corresponding Athena APIs. Not all APIs support all integration patterns, as shown in the following table. | API | Request Response | Run a Job (.sync) | | --- | --- | --- | | StartQueryExecution | Supported | Supported | | StopQueryExecution | Supported | Not supported | | GetQueryExecution | Supported | Not supported | | GetQueryResults | Supported | Not supported | The following includes a Task state that starts an Athena query. ``` "Start an Athena query": { "Type": "Task", "Resource": "arn:aws:states:::athena:startQueryExecution.sync", "Arguments": { "QueryString": "SELECT * FROM \"myDatabase\".\"myTable\" limit 1", "WorkGroup": "primary", "ResultConfiguration": { "OutputLocation": "s3://amzn-s3-demo-bucket" } }, "Next": "Get results of the query" } ``` ## Optimized Amazon Athena APIs: + [https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html) + [https://docs.aws.amazon.com/athena/latest/APIReference/API_StopQueryExecution.html](https://docs.aws.amazon.com/athena/latest/APIReference/API_StopQueryExecution.html) + [https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html) + [https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html) **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). ## IAM policies for calling Amazon Athena The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). **Note** In addition to IAM policies, you might need to use AWS Lake Formation to grant access to data in services, such as Amazon S3 and the AWS Glue Data Catalog. For more information, see [Use Athena to query data registered with AWS Lake Formation](https://docs.aws.amazon.com/athena/latest/ug/security-athena-lake-formation.html). ### `StartQueryExecution` *Static resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:startQueryExecution", "athena:stopQueryExecution", "athena:getQueryExecution", "athena:getDataCatalog", "athena:GetWorkGroup", "athena:BatchGetQueryExecution", "athena:GetQueryResults", "athena:ListQueryExecutions" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/myWorkGroup", "arn:aws:athena:us-east-1:123456789012:datacatalog/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload", "s3:CreateBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*" ] }, { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:GetDatabase", "glue:GetDatabases", "glue:UpdateDatabase", "glue:DeleteDatabase", "glue:CreateTable", "glue:UpdateTable", "glue:GetTable", "glue:GetTables", "glue:DeleteTable", "glue:BatchDeleteTable", "glue:BatchCreatePartition", "glue:CreatePartition", "glue:UpdatePartition", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition", "glue:DeletePartition", "glue:BatchDeletePartition" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:catalog", "arn:aws:glue:us-east-1:123456789012:database/*", "arn:aws:glue:us-east-1:123456789012:table/*", "arn:aws:glue:us-east-1:123456789012:userDefinedFunction/*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:startQueryExecution", "athena:getDataCatalog" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/myWorkGroup", "arn:aws:athena:us-east-1:123456789012:datacatalog/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload", "s3:CreateBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*" ] }, { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:GetDatabase", "glue:GetDatabases", "glue:UpdateDatabase", "glue:DeleteDatabase", "glue:CreateTable", "glue:UpdateTable", "glue:GetTable", "glue:GetTables", "glue:DeleteTable", "glue:BatchDeleteTable", "glue:BatchCreatePartition", "glue:CreatePartition", "glue:UpdatePartition", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition", "glue:DeletePartition", "glue:BatchDeletePartition" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:catalog", "arn:aws:glue:us-east-1:123456789012:database/*", "arn:aws:glue:us-east-1:123456789012:table/*", "arn:aws:glue:us-east-1:123456789012:userDefinedFunction/*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] } ``` ------ *Dynamic resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:startQueryExecution", "athena:stopQueryExecution", "athena:getQueryExecution", "athena:getDataCatalog", "athena:GetWorkGroup", "athena:BatchGetQueryExecution", "athena:GetQueryResults", "athena:ListQueryExecutions" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/*", "arn:aws:athena:us-east-1:123456789012:datacatalog/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload", "s3:CreateBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*" ] }, { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:GetDatabase", "glue:GetDatabases", "glue:UpdateDatabase", "glue:DeleteDatabase", "glue:CreateTable", "glue:UpdateTable", "glue:GetTable", "glue:GetTables", "glue:DeleteTable", "glue:BatchDeleteTable", "glue:BatchCreatePartition", "glue:CreatePartition", "glue:UpdatePartition", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition", "glue:DeletePartition", "glue:BatchDeletePartition" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:catalog", "arn:aws:glue:us-east-1:123456789012:database/*", "arn:aws:glue:us-east-1:123456789012:table/*", "arn:aws:glue:us-east-1:123456789012:userDefinedFunction/*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:startQueryExecution", "athena:getDataCatalog" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/*", "arn:aws:athena:us-east-1:123456789012:datacatalog/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload", "s3:CreateBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::*" ] }, { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:GetDatabase", "glue:GetDatabases", "glue:UpdateDatabase", "glue:DeleteDatabase", "glue:CreateTable", "glue:UpdateTable", "glue:GetTable", "glue:GetTables", "glue:DeleteTable", "glue:BatchDeleteTable", "glue:BatchCreatePartition", "glue:CreatePartition", "glue:UpdatePartition", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition", "glue:DeletePartition", "glue:BatchDeletePartition" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:catalog", "arn:aws:glue:us-east-1:123456789012:database/*", "arn:aws:glue:us-east-1:123456789012:table/*", "arn:aws:glue:us-east-1:123456789012:userDefinedFunction/*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] } ``` ------ ### `StopQueryExecution` *Resources* **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:stopQueryExecution" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/*" ] } ] } ``` ### `GetQueryExecution` *Resources* **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:getQueryExecution" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/*" ] } ] } ``` ### `GetQueryResults` *Resources* **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect": "Allow", "Action": [ "athena:getQueryResults" ], "Resource": [ "arn:aws:athena:us-east-1:123456789012:workgroup/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::*" ] } ] } ``` # Run AWS Batch workloads with Step Functions You can integrate Step Functions with AWS Batch to run batch computing workloads in the AWS cloud. This page lists the supported AWS Batch APIs and provides an example `Task` state to perform a batch-processing task. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized AWS Batch integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is available. Note that there are no specific optimizations for the [Request Response](connect-to-resource.md#connect-default) or [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration patterns. The following shows an example `Task` state that submits an AWS Batch job and waits for it to complete. Many of the arguments shown are optional. ``` "Submit Batch Job": { "Type": "Task", "Resource": "arn:aws:states:::batch:submitJob.sync", "Arguments": { "JobName": "BATCH_NAME", "JobQueue": "BATCH_QUEUE_ARN", "JobDefinition": "BATCH_JOB_DEFINITION_ARN", "ArrayProperties": { "Size": 10 }, "ContainerOverrides": { "ResourceRequirements": [ { "Type": "VCPU", "Value": "4" } ] }, "DependsOn": [ { "JobId": "myJobId", "Type": "SEQUENTIAL" } ], "PropagateTags": true, "Arguments": { "Key1": "value1", "Key2": 100 }, "RetryStrategy": { "Attempts": 1 }, "Tags": { "Tag": "TAG" }, "Timeout": { "AttemptDurationSeconds": 10 } } } ``` ## Optimized AWS Batch APIs: + [https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html) **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## IAM policies for calling AWS Batch The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). Because job ids for `SubmitJob` and `TerminateJob` are generated and therefore only known at runtime, you cannot create a policy that restricts access based on a specific resource. **Tip for fine grained access** To add fine grained access to `SubmitJob` and `TerminateJob`, consider using tags for jobs and creating a policy that limits access based on your tags. In addition, the job queue, definition, and consumable resources can be restricted for `SubmitJob` using known resources. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "batch:SubmitJob", "batch:DescribeJobs", "batch:TerminateJob" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForBatchJobsRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "batch:SubmitJob" ], "Resource": "*" } ] } ``` ------ # Invoke and customize Amazon Bedrock models with Step Functions You can integrate Step Functions with Amazon Bedrock to invoke a specified Amazon Bedrock model and create a fine-tuning job to customize a model. This page lists the optimized Amazon Bedrock APIs and provides an example `Task` state to extract the result of a model invocation. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Tip** To deploy an example workflow that integrates with Amazon Bedrock, see [Perform AI prompt-chaining with Amazon Bedrock](sample-bedrock-prompt-chaining.md). ## Amazon Bedrock service integration APIs To integrate AWS Step Functions with Amazon Bedrock, you can use the following APIs. These APIs are similar to the corresponding Amazon Bedrock APIs, except *InvokeModel* has additional request fields. **Amazon Bedrock API - [CreateModelCustomizationJob](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html)** Creates a fine-tuning job to customize a base model. You can invoke the Step Functions integration API with **CreateModelCustomizationJob** for *Request Response*, or **CreateModelCustomizationJob.sync** for *Run a Job (.sync)* integration patterns. There are no differences in the fields for the API calls. **Amazon Bedrock API - [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html)** Invokes the specified Amazon Bedrock model to run inference using the input you provide in the request body. You use `InvokeModel` to run inference for text models, image models, and embedding models. The Amazon Bedrock service integration API request body for *InvokeModel* includes the following additional parameters. + `Body` – Specifies input data in the format specified in the content-type request header. `Body` contains parameters specific to the target model. If you use the `InvokeModel` API, you must specify the `Body` parameter. Step Functions doesn't validate the input you provide in `Body`. When you specify `Body` using the Amazon Bedrock optimized integration, you can specify a payload of up to 256 KiB. If your payload exceeds 256 KiB, we recommend that you use `Input`. + `Input` – Specifies the source to retrieve the input data from. This optional field is specific to Amazon Bedrock optimized integration with Step Functions. In this field, you can specify an `S3Uri`. You can specify either `Body` in the Parameters or `Input`, but not both. When you specify `Input` without specifying `ContentType`, the content type of the input data source becomes the value for `ContentType`. + `Output` – Specifies the destination where the API response is written. This optional field is specific to Amazon Bedrock optimized integration with Step Functions. In this field, you can specify an `S3Uri`. If you specify this field, the API response body is replaced with a reference to the Amazon S3 location of the original output. The following example shows the syntax for InvokeModel API for Amazon Bedrock integration. ``` { "ModelId": String, // required "Accept": String, // default: application/json "ContentType": String, // default: application/json "Input": { // not from Bedrock API "S3Uri": String }, "Output": { // not from Bedrock API "S3Uri": String } } ``` ## Task state definition for Amazon Bedrock integration The following Task state definition shows how you can integrate with Amazon Bedrock in your state machines. This example shows a Task state that extracts the full result of model invocation specified by the path, `result_one`. This is based on [Inference parameters for foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html). This example uses the Cohere Command large language model (LLM). ``` { "Type": "Task", "Resource": "arn:aws:states:::bedrock:invokeModel", "Arguments": { "ModelId": "cohere.command-text-v14", "Body": { "prompt": "{% states.input.prompt_one %}", "max_tokens": 20 }, "ContentType": "application/json", "Accept": "*/*" }, "End": true } ``` ## IAM policies for calling Amazon Bedrock When you create a state machine using the console, Step Functions automatically creates an execution role for your state machine with the least privileges required. These automatically generated IAM roles are valid for the AWS Region in which you create the state machine. We recommend that when you create IAM policies, do not include wildcards in the policies. As a security best practice, you should scope your policies down as much as possible. You should use dynamic policies only when certain input parameters are not known during runtime. The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). ### IAM policy examples for Amazon Bedrock integration The following section describes the IAM permissions you need based on the Amazon Bedrock API that you use for a specific foundation or provisioned model. This section also contains examples of policies that grant full access. Remember to replace the *italicized* text with your resource-specific information. + [IAM policy example to access a specific foundation model using InvokeModel](#bedrock-policy-invoke-foundation-model) + [IAM policy example to access a specific provisioned model using InvokeModel](#bedrock-policy-invoke-provisioned-model) + [Full access IAM policy example to use InvokeModel](#bedrock-policy-invokemodel-full-access) + [IAM policy example to access a specific foundation model as a base model](#bedrock-policy-foundation-model) + [IAM policy example to access a specific custom model as a base model](#bedrock-policy-custom-model) + [Full access IAM policy example to use CreateModelCustomizationJob.sync](#bedrock-policy-createmodel-full-access) + [IAM policy example to access a specific foundation model using CreateModelCustomizationJob.sync](#bedrock-policy-createmodel-sync-foundation-model) + [IAM policy example to access a custom model using CreateModelCustomizationJob.sync](#bedrock-policy-createmodel-sync-custom-model) + [Full access IAM policy example to use CreateModelCustomizationJob.sync](#bedrock-policy-createmodel-sync-full-access) #### IAM policy example to access a specific foundation model using InvokeModel The following is an IAM policy example for a state machine that accesses a specific foundation model named `amazon.titan-text-express-v1` using the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "InvokeModel1", "Action": [ "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-text-express-v1" ] } ] } ``` #### IAM policy example to access a specific provisioned model using InvokeModel The following is an IAM policy example for a state machine that accesses a specific provisioned model named `c2oi931ulksx` using the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "InvokeModel1", "Action": [ "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:us-east-1:123456789012:provisioned-model/c2oi931ulksx" ] } ] } ``` #### Full access IAM policy example to use InvokeModel The following is an IAM policy example for a state machine that provides full access when you use the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "InvokeModel1", "Action": [ "bedrock:InvokeModel" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/*", "arn:aws:bedrock:us-east-1:123456789012:provisioned-model/*" ] } ] } ``` #### IAM policy example to access a specific foundation model as a base model The following is an IAM policy example for a state machine to access a specific foundation model named `amazon.titan-text-express-v1` as a base model using the [CreateModelCustomizationJob](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "CreateModelCustomizationJob1", "Action": [ "bedrock:CreateModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-text-express-v1", "arn:aws:bedrock:us-east-1:123456789012:custom-model/*", "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob2", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRole" ] } ] } ``` #### IAM policy example to access a specific custom model as a base model The following is an IAM policy example for a state machine to access a specific custom model as a base model using the [CreateModelCustomizationJob](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "CreateModelCustomizationJob1", "Action": [ "bedrock:CreateModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1:123456789012:custom-model/*", "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob2", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRoleName" ] } ] } ``` #### Full access IAM policy example to use CreateModelCustomizationJob.sync The following is an IAM policy example for a state machine that provides full access when you use the [CreateModelCustomizationJob](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "CreateModelCustomizationJob1", "Action": [ "bedrock:CreateModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/*", "arn:aws:bedrock:us-east-1:123456789012:custom-model/*", "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob2", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRole" ] } ] } ``` #### IAM policy example to access a specific foundation model using CreateModelCustomizationJob.sync The following is an IAM policy example for a state machine to access a specific foundation model named `amazon.titan-text-express-v1` using the [CreateModelCustomizationJob.sync](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "CreateModelCustomizationJob1", "Action": [ "bedrock:CreateModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-text-express-v1", "arn:aws:bedrock:us-east-1:123456789012:custom-model/*", "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob2", "Action": [ "bedrock:GetModelCustomizationJob", "bedrock:StopModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob3", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRole" ] } ] } ``` #### IAM policy example to access a custom model using CreateModelCustomizationJob.sync The following is an IAM policy example for a state machine to access a custom model using the [CreateModelCustomizationJob.sync](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "CreateModelCustomizationJob1", "Action": [ "bedrock:CreateModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1:123456789012:custom-model/*", "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob2", "Action": [ "bedrock:GetModelCustomizationJob", "bedrock:StopModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob3", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRole" ] } ] } ``` #### Full access IAM policy example to use CreateModelCustomizationJob.sync The following is an IAM policy example for a state machine that provides full access when you use the [CreateModelCustomizationJob.sync](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelCustomizationJob.html) API action. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Sid": "CreateModelCustomizationJob1", "Action": [ "bedrock:CreateModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/*", "arn:aws:bedrock:us-east-1:123456789012:custom-model/*", "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob2", "Action": [ "bedrock:GetModelCustomizationJob", "bedrock:StopModelCustomizationJob" ], "Resource": [ "arn:aws:bedrock:us-east-1:123456789012:model-customization-job/*" ] }, { "Effect": "Allow", "Sid": "CreateModelCustomizationJob3", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRole" ] } ] } ``` # Manage AWS CodeBuild builds with Step Functions You can integrate Step Functions with AWS CodeBuild to start, stop, and manage builds. This page lists the supported CodeBuild APIs you can use with Step Functions. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). With the Step Functions integration with AWS CodeBuild you can use Step Functions to trigger, stop, and manage builds, and to share build reports. Using Step Functions, you can design and run continuous integration pipelines for validating your software changes for applications. **Key features of Optimized CodeBuild integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. After you call `StopBuild` or `StopBuildBatch`, the build or build batch is not immediately deletable until some internal work is completed within CodeBuild to finalize the state of the build or builds. If you attempt to use `BatchDeleteBuilds` or `DeleteBuildBatch` during this period, the build or build batch may not be deleted. The optimized service integrations for `BatchDeleteBuilds` and `DeleteBuildBatch` include an internal retry to simplify the use case of deleting immediately after stopping. Not all APIs support all integration patterns, as shown in the following table. | API | Request Response | Run a Job (.sync) | | --- | --- | --- | | StartBuild | Supported | Supported | | StopBuild | Supported | Not supported | | BatchDeleteBuilds | Supported | Not supported | | BatchGetReports | Supported | Not supported | | StartBuildBatch | Supported | Supported | | StopBuildBatch | Supported | Not supported | | RetryBuildBatch | Supported | Supported | | DeleteBuildBatch | Supported | Not supported | **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## Optimized CodeBuild APIs + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StartBuild.html#API_StartBuild_RequestSyntax](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StartBuild.html#API_StartBuild_RequestSyntax) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StopBuild.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StopBuild.html) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_BatchDeleteBuilds.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_BatchDeleteBuilds.html) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_BatchGetReports.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_BatchGetReports.html) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StartBuildBatch.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StartBuildBatch.html) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StopBuildBatch.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_StopBuildBatch.html) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_RetryBuildBatch.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_RetryBuildBatch.html) + [https://docs.aws.amazon.com/codebuild/latest/APIReference/API_DeleteBuildBatch.html](https://docs.aws.amazon.com/codebuild/latest/APIReference/API_DeleteBuildBatch.html) **Note** When using JSONPath, you can use the recursive descent operator (`..`) to provide parameters for `BatchDeleteBuilds`. With the returned array, you can transform the `Arn` field from `StartBuild` into a plural `Ids` parameter, as shown in the following example. ``` "BatchDeleteBuilds": { "Type": "Task", "Resource": "arn:aws:states:::codebuild:batchDeleteBuilds", "Arguments": { "Ids.$": "$.Build..Arn" }, "Next": "MyNextState" }, ``` ## IAM policies for calling AWS CodeBuild The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). *Resources*: **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "sns:Publish" ], "Resource": [ "arn:aws:sns:us-east-1:123456789012:StepFunctionsSample-CodeBuildExecution1111-2222-3333-wJalrXUtnFEMI-SNSTopic-bPxRfiCYEXAMPLEKEY" ], "Effect": "Allow" }, { "Action": [ "codebuild:StartBuild", "codebuild:StopBuild", "codebuild:BatchGetBuilds", "codebuild:BatchGetReports" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventForCodeBuildStartBuildRule" ], "Effect": "Allow" } ] } ``` ### `StartBuild` *Static resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuild", "codebuild:StopBuild", "codebuild:BatchGetBuilds" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventForCodeBuildStartBuildRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuild" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` ------ *Dynamic resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuild", "codebuild:StopBuild", "codebuild:BatchGetBuilds" ], "Resource": [ "arn:aws:codebuild:us-east-1:*:project/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventForCodeBuildStartBuildRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuild" ], "Resource": [ "arn:aws:codebuild:us-east-1:*:project/*" ] } ] } ``` ------ ### `StopBuild` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StopBuild" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StopBuild" ], "Resource": [ "arn:aws:codebuild:us-east-1:*:project/*" ] } ] } ``` ### `BatchDeleteBuilds` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:BatchDeleteBuilds" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:BatchDeleteBuilds" ], "Resource": [ "arn:aws:codebuild:us-east-1:*:project/*" ] } ] } ``` ### `BatchGetReports` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:BatchGetReports" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:report-group/myReportName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:BatchGetReports" ], "Resource": [ "arn:aws:codebuild:us-east-1:*:report-group/*" ] } ] } ``` ### `StartBuildBatch` *Static resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuildBatch", "codebuild:StopBuildBatch", "codebuild:BatchGetBuildBatches" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventForCodeBuildStartBuildBatchRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` ------ *Dynamic resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuildBatch", "codebuild:StopBuildBatch", "codebuild:BatchGetBuildBatches" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventForCodeBuildStartBuildBatchRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StartBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/*" ] } ] } ``` ------ ### `StopBuildBatch` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StopBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:StopBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/*" ] } ] } ``` ### `RetryBuildBatch` *Static resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:RetryBuildBatch", "codebuild:StopBuildBatch", "codebuild:BatchGetBuildBatches" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:RetryBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` ------ *Dynamic resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:RetryBuildBatch", "codebuild:StopBuildBatch", "codebuild:BatchGetBuildBatches" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/*" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:RetryBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/*" ] } ] } ``` ------ ### `DeleteBuildBatch` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:DeleteBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/myProjectName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "codebuild:DeleteBuildBatch" ], "Resource": [ "arn:aws:codebuild:us-east-1:123456789012:project/*" ] } ] } ``` # Perform DynamoDB CRUD operations with Step Functions You can integrate Step Functions with DynamoDB to perform CRUD operations on a DynamoDB table. This page lists the supported DynamoDB APIs and provides an example `Task` state to retrieve an item from DynamoDB. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of optimized DynamoDB integration** There is no specific optimization for the [Request Response](connect-to-resource.md#connect-default) integration pattern. [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration pattern is not supported. Only [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html), [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html), [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html), and [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html) API actions are available through optimized integration. Other API actions, such as [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html) are available using the DynamoDB AWS SDK integration. The following is an example `Task` state that retrieves a message from DynamoDB. ``` "Read next Message from DynamoDB": { "Type": "Task", "Resource": "arn:aws:states:::dynamodb:getItem", "Arguments": { "TableName": "DYNAMO_DB_TABLE_NAME", "Key": { "MessageId": {"S": "{% $List[0] %}"} } } ``` To see this state in a working example, see the [Transfer data records with Lambda, DynamoDB, and Amazon SQS](sample-project-transfer-data-sqs.md) starter template. **Exception prefix differences** When standard DynamoDB connections experience an error, the exception prefix will be `DynamoDb` (mixed case). For optimized integrations, the exception prefix will be `DynamoDB` (uppercase `DB`). **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). ## Optimized DynamoDB APIs + [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html) + [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html) + [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html) + [https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html) **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## IAM policies for calling DynamoDB The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem" ], "Resource": [ "arn:aws:dynamodb:us-east-1:123456789012:table/myTableName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem" ], "Resource": "*" } ] } ``` For more information about the IAM policies for all DynamoDB API actions, see [IAM policies with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/using-identity-based-policies.html) in the *Amazon DynamoDB Developer Guide*. Additionally, for information about the IAM policies for PartiQL for DynamoDB, see [IAM policies with PartiQL for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ql-iam.html) in the *Amazon DynamoDB Developer Guide*. # Run Amazon ECS or Fargate tasks with Step Functions Learn how to integrate Step Functions with Amazon ECS or Fargate to run and manage tasks. In Amazon ECS, a task is the fundamental unit of computation. Tasks are defined by a task definition that specifies how a Docker container should be run, including the container image, CPU and memory limits, network configuration, and other parameters. This page lists the available Amazon ECS API actions and provides instructions on how to pass data to an Amazon ECS task using Step Functions. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized Amazon ECS/Fargate integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. `ecs:runTask` can return an HTTP 200 response, but have a non-empty `Failures` field as follows: **Request Response**: Return the response and do not fail the task, which is the same as non-optimized integrations. **Run a Job or Task Token**: If a non-empty `Failures` field is encountered, the task is failed with an `AmazonECS.Unknown` error. ## Optimized Amazon ECS/Fargate APIs + [https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html) starts a new task using the specified task definition. **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## Passing Data to an Amazon ECS Task To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). You can use `overrides` to override the default command for a container, and pass input to your Amazon ECS tasks. See [https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html). In the example, we have used JsonPath to pass values to the `Task` from the input to the `Task` state. The following includes a `Task` state that runs an Amazon ECS task and waits for it to complete. ``` { "StartAt": "Run an ECS Task and wait for it to complete", "States": { "Run an ECS Task and wait for it to complete": { "Type": "Task", "Resource": "arn:aws:states:::ecs:runTask.sync", "Arguments": { "Cluster": "cluster-arn", "TaskDefinition": "job-id", "Overrides": { "ContainerOverrides": [ { "Name": "container-name", "Command": "{% $state.input.commands %}" } ] } }, "End": true } } } ``` The `Command` line in `ContainerOverrides` passes the commands from the state input to the container. In the previous example state machine, given the following input, each of the commands would be passed as a container override: ``` { "commands": [ "test command 1", "test command 2", "test command 3" ] } ``` The following includes a `Task` state that runs an Amazon ECS task, and then waits for the task token to be returned. See [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token). ``` { "StartAt":"Manage ECS task", "States":{ "Manage ECS task":{ "Type":"Task", "Resource":"arn:aws:states:::ecs:runTask.waitForTaskToken", "Arguments":{ "LaunchType":"FARGATE", "Cluster":"cluster-arn", "TaskDefinition":"job-id", "Overrides":{ "ContainerOverrides":[ { "Name":"container-name", "Environment":[ { "Name" : "TASK_TOKEN_ENV_VARIABLE", "Value" : "{% $states.context.Task.Token %}" } ] } ] } }, "End":true } } } ``` ## IAM policies for calling Amazon ECS/AWS Fargate The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). Because the value for `TaskId` is not known until the task is submitted, Step Functions creates a more privileged `"Resource": "*"` policy. **Note** You can only stop Amazon Elastic Container Service (Amazon ECS) tasks that were started by Step Functions, despite the `"*"` IAM policy. ------ #### [ Run a Job (.sync) ] *Static resources* ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:RunTask" ], "Resource": [ "arn:aws:ecs:region: account-id:task-definition/taskDefinition:revisionNumber" ] }, { "Effect": "Allow", "Action": [ "ecs:StopTask", "ecs:DescribeTasks" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:region: account-id:rule/StepFunctionsGetEventsForECSTaskRule" ] } ] } ``` *Dynamic resources* ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:RunTask", "ecs:StopTask", "ecs:DescribeTasks" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:region: account-id:rule/StepFunctionsGetEventsForECSTaskRule" ] } ] } ``` ------ #### [ Request Response and Callback (.waitForTaskToken) ] *Static resources* ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:RunTask" ], "Resource": [ "arn:aws:ecs:region: account-id:task-definition/taskDefinition:revisionNumber" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecs:RunTask" ], "Resource": "*" } ] } ``` ------ If your scheduled Amazon ECS tasks require the use of a task execution role, a task role, or a task role override, then you must add `iam:PassRole` permissions for each task execution role, task role, or task role override to the CloudWatch Events IAM role of the calling entity, which in this case is Step Functions. # Create and manage Amazon EKS clusters with Step Functions Learn how to integrate Step Functions with Amazon EKS to manage Kubernetes clusters. Step Functions provides two types of service integration APIs for integrating with Amazon Elastic Kubernetes Service. One lets you use the Amazon EKS APIs to create and manage an Amazon EKS cluster. The other lets you interact with your cluster using the Kubernetes API and run jobs as part of your application’s workflow. You can use the Kubernetes API integrations with Amazon EKS clusters created using Step Functions, with Amazon EKS clusters created by the **eksctl** tool or the [Amazon EKS console](https://console.aws.amazon.com/eks/home), or similar methods. For more information, see [Creating an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) in the Amazon EKS User Guide. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized Amazon EKS integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. There are no specific optimizations for the [Request Response](connect-to-resource.md#connect-default) integration pattern. The [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration pattern is not supported. **Note** The Step Functions EKS integration supports only Kubernetes APIs with public endpoint access. By default, EKS clusters API server endpoints have public access. For more information, see [Amazon EKS cluster endpoint access control](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) in the **Amazon EKS User Guide**. Step Functions does not terminate an Amazon EKS cluster automatically if execution is stopped. If your state machine stops before your Amazon EKS cluster has terminated, your cluster may continue running indefinitely, and can accrue additional charges. To avoid this, ensure that any Amazon EKS cluster you create is terminated properly. For more information, see: + [Deleting a cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) in the Amazon EKS User Guide. + [Run a Job (.sync)](connect-to-resource.md#connect-sync) in Service Integration Patterns. **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). ## Kubernetes API integrations Step Functions supports the following Kubernetes APIs: ### RunJob The `eks:runJob` service integration allows you to run a job on your Amazon EKS cluster. The `eks:runJob.sync` variant allows you to wait for the job to complete, and, optionally retrieve logs. Your Kubernetes API server must grant permissions to the IAM role used by your state machine. For more information, see [Permissions](#connect-eks-permissions). For the **Run a Job** (`.sync`) pattern, the status of the job is determined by polling. Step Functions initially polls at a rate of approximately 1 poll per minute. This rate eventually slows to approximately 1 poll every 5 minutes. If you require more frequent polling, or require more control over the polling strategy, you can use the `eks:call` integration to query the status of the job. The `eks:runJob` integration is specific to `batch/v1` Kubernetes Jobs. For more information, see [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) in the Kubernetes documentation. If you want to manage other Kubernetes resources, including custom resources, use the `eks:call` service integration. You can use Step Functions to build polling loops, as demonstrated in the [Poll for job status with Lambda and AWS Batch](sample-project-job-poller.md) sample project. Supported parameters include: + `ClusterName`: The name of the Amazon EKS cluster you want to call. + `Type`: `String` + Required: yes + `CertificateAuthority`: The Base64-encoded certificate data required to communicate with your cluster. You can obtain this value from the [Amazon EKS console](https://console.aws.amazon.com/eks/home) or by using the Amazon EKS [DescribeCluster](https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeCluster.html) API. + `Type`: `String` + Required: yes + `Endpoint`: The endpoint URL for your Kubernetes API server. You can obtain this value from the [Amazon EKS console](https://console.aws.amazon.com/eks/home) or by using the Amazon EKS [DescribeCluster](https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeCluster.html) API. + `Type`: `String` + Required: yes + `Namespace`: The namespace in which to run the job. If not provided, the namespace `default` is used. + `Type`: `String` + Required: no + `Job`: The definition of the Kubernetes Job. See [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) in the Kubernetes documentation. + `Type`: `JSON` or `String` + Required: yes + `LogOptions`: A set of options to control the optional retrieval of logs. Only applicable if the Run a Job (.sync) service integration pattern is used to wait for the completion of the job. + `Type`: `JSON` + Required: no + Logs are included in the response under the key `logs`. There may be multiple pods within the job, each with multiple containers. ``` { ... "logs": { "pods": { "pod1": { "containers": { "container1": { "log": }, ... } }, ... } } ``` + Log retrieval is performed on a best-effort basis. If there is an error retrieving a log, in place of the `log` field there will be the fields `error` and `cause`. + `LogOptions.RetrieveLogs`: Enable log retrieval after the job completes. By default, logs are not retrieved. + `Type`: `Boolean` + Required: no + `LogOptions.RawLogs`: If `RawLogs` is set to true, logs will be returned as raw strings without attempting to parse them into JSON. By default, logs are deserialized into JSON if possible. In some cases such parsing can introduce unwanted changes, such as limiting the precision of numbers containing many digits. + `Type`: `Boolean` + Required: no + `LogOptions.LogParameters`: The Kubernetes API’s Read Log API supports query parameters to control log retrieval. For example, you can use `tailLines` or `limitBytes` to limit the size of retrieved logs and remain within the Step Functions data size quota. For more information, see the [Read Log](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#read-log-pod-v1-core) section of the Kubernetes API Reference. + `Type: `Map of `String` to `List of Strings` + Required: no + Example: ``` "LogParameters": { "tailLines": [ "6" ] } ``` The following example includes a `Task` state that runs a job, waits for it to complete, then retrieves the job’s logs: ``` { "StartAt": "Run a job on EKS", "States": { "Run a job on EKS": { "Type": "Task", "Resource": "arn:aws:states:::eks:runJob.sync", "Arguments": { "ClusterName": "MyCluster", "CertificateAuthority": "ANPAJ2UCCR6DPCEXAMPLE", "Endpoint": "https://AKIAIOSFODNN7EXAMPLE.yl4.us-east-1.eks.amazonaws.com", "LogOptions": { "RetrieveLogs": true }, "Job": { "apiVersion": "batch/v1", "kind": "Job", "metadata": { "name": "example-job" }, "spec": { "backoffLimit": 0, "template": { "metadata": { "name": "example-job" }, "spec": { "containers": [ { "name": "pi-2000", "image": "perl", "command": [ "perl" ], "args": [ "-Mbignum=bpi", "-wle", "print bpi(2000)" ] } ], "restartPolicy": "Never" } } } } }, "End": true } } } ``` ### `Call` The `eks:call` service integration allows you to use the Kubernetes API to read and write Kubernetes resource objects via a Kubernetes API endpoint. Your Kubernetes API server must grant permissions to the IAM role used by your state machine. For more information, see [Permissions](#connect-eks-permissions). For more information about the available operations, see the [Kubernetes API Reference](https://kubernetes.io/docs/reference/kubernetes-api/). Supported parameters for `Call` include: + `ClusterName`: The name of the Amazon EKS cluster you want to call. + `Type`: String + Required: Yes + `CertificateAuthority`: The Base64-encoded certificate data required to communicate with your cluster. You can obtain this value from the [Amazon EKS console](https://console.aws.amazon.com/eks/home) or by using the Amazon EKS [DescribeCluster](https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeCluster.html) API. + `Type`: `String` + Required: Yes + `Endpoint`: The endpoint URL for your Kubernetes API server. You can find this value on the [Amazon EKS console](https://console.aws.amazon.com/eks/home) or by using Amazon EKS’ DescribeCluster API. + `Type`: `String` + Required: Yes + `Method`: The HTTP method of your request. One of: `GET`, `POST`, `PUT`, `DELETE`, `HEAD`, or `PATCH`. + `Type`: `String` + Required: Yes + `Path`: The HTTP path of the Kubernetes REST API operation. + `Type`: `String` + Required: Yes + `QueryParameters`: The HTTP query parameters of the Kubernetes REST API operation. + `Type: `Map of `String` to `List of Strings` + Required: No + Example: ``` "QueryParameters": { "labelSelector": [ "job-name=example-job" ] } ``` + `RequestBody`: The HTTP message body of the Kubernetes REST API operation. + `Type`: `JSON` or `String` + Required: No The following includes a `Task` state that uses `eks:call` to list the pods belonging to the job `example-job`. ``` { "StartAt": "Call EKS", "States": { "Call EKS": { "Type": "Task", "Resource": "arn:aws:states:::eks:call", "Arguments": { "ClusterName": "MyCluster", "CertificateAuthority": "ANPAJ2UCCR6DPCEXAMPLE", "Endpoint": "https://444455556666.yl4.us-east-1.eks.amazonaws.com", "Method": "GET", "Path": "/api/v1/namespaces/default/pods", "QueryParameters": { "labelSelector": [ "job-name=example-job" ] } }, "End": true } } } ``` The following includes a `Task` state that uses `eks:call` to delete the job `example-job`, and sets the `propagationPolicy` to ensure the job's pods are also deleted. ``` { "StartAt": "Call EKS", "States": { "Call EKS": { "Type": "Task", "Resource": "arn:aws:states:::eks:call", "Arguments": { "ClusterName": "MyCluster", "CertificateAuthority": "ANPAJ2UCCR6DPCEXAMPLE", "Endpoint": "https://444455556666.yl4.us-east-1.eks.amazonaws.com", "Method": "DELETE", "Path": "/apis/batch/v1/namespaces/default/jobs/example-job", "QueryParameters": { "propagationPolicy": [ "Foreground" ] } }, "End": true } } } ``` ## Optimized Amazon EKS APIs Supported Amazon EKS APIs and syntax include: + [https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateCluster.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateCluster.html) + [Request syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateCluster.html#API_CreateCluster_RequestSyntax) + [Response syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateCluster.html#API_CreateCluster_ResponseSyntax) When an Amazon EKS cluster is created using the `eks:createCluster` service integration, the IAM role is added to the Kubernetes RBAC authorization table as the administrator (with system:masters permissions). Initially, only that IAM entity can make calls to the Kubernetes API server. For more information, see: + [Managing users or IAM roles for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) in the *Amazon EKS User Guide* + The [Permissions](#connect-eks-permissions) section Amazon EKS uses service-linked roles which contain the permissions Amazon EKS requires to call other services on your behalf. If these service-linked roles do not exist in your account already, you must add the `iam:CreateServiceLinkedRole` permission to the IAM role used by Step Functions. For more information, see [Using Service-Linked Roles](https://docs.aws.amazon.com/eks/latest/userguide/using-service-linked-roles.html) in the *Amazon EKS User Guide*. The IAM role used by Step Functions must have `iam:PassRole` permissions to pass the cluster IAM role to Amazon EKS. For more information, see [Amazon EKS cluster IAM role](https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html) in the *Amazon EKS User Guide*. + [https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteCluster.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteCluster.html) + [Request syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteCluster.html#API_DeleteCluster_RequestSyntax) + [Response syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteCluster.html#API_DeleteCluster_ResponseSyntax) You must delete any Fargate profiles or node groups before deleting a cluster. + [https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateFargateProfile.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateFargateProfile.html) + [Request syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateFargateProfile.html#API_CreateFargateProfile_RequestSyntax) + [Response syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateFargateProfile.html#API_CreateFargateProfile_ResponseSyntax) Amazon EKS uses service-linked roles which contain the permissions Amazon EKS requires to call other services on your behalf. If these service-linked roles do not exist in your account already, you must add the `iam:CreateServiceLinkedRole` permission to the IAM role used by Step Functions. For more information, see [Using Service-Linked Roles](https://docs.aws.amazon.com/eks/latest/userguide/using-service-linked-roles.html) in the *Amazon EKS User Guide*. Amazon EKS on Fargate may not be available in all regions. For information on region availability, see the section on [Fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html) in the *Amazon EKS User Guide*. The IAM role used by Step Functions must have `iam:PassRole` permissions to pass the pod execution IAM role to Amazon EKS. For more information, see [Pod execution role](https://docs.aws.amazon.com/eks/latest/userguide/pod-execution-role.html) in the *Amazon EKS User Guide*. + [https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteFargateProfile.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteFargateProfile.html) + [Request syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteFargateProfile.html#API_DeleteFargateProfile_RequestSyntax) + [Response syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteFargateProfile.html#API_DeleteFargateProfile_ResponseSyntax) + [https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateNodegroup.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateNodegroup.html) + [Request syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateNodegroup.html#API_CreateNodegroup_RequestSyntax) + [Response syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateNodegroup.html#API_CreateNodegroup_ResponseSyntax) Amazon EKS uses service-linked roles which contain the permissions Amazon EKS requires to call other services on your behalf. If these service-linked roles do not exist in your account already, you must add the `iam:CreateServiceLinkedRole` permission to the IAM role used by Step Functions. For more information, see [Using Service-Linked Roles](https://docs.aws.amazon.com/eks/latest/userguide/using-service-linked-roles.html) in the *Amazon EKS User Guide*. The IAM role used by Step Functions must have `iam:PassRole` permissions to pass the node IAM role to Amazon EKS. For more information, see [Using Service-Linked Roles](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) in the *Amazon EKS User Guide*. + [https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteNodegroup.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteNodegroup.html) + [Request syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteNodegroup.html#API_DeleteNodegroup_RequestSyntax) + [Response syntax](https://docs.aws.amazon.com/eks/latest/APIReference/API_DeleteNodegroup.html#API_DeleteNodegroup_ResponseSyntax) The following includes a `Task` that creates an Amazon EKS cluster. ``` { "StartAt": "CreateCluster.sync", "States": { "CreateCluster.sync": { "Type": "Task", "Resource": "arn:aws:states:::eks:createCluster.sync", "Arguments": { "Name": "MyCluster", "ResourcesVpcConfig": { "SubnetIds": [ "subnet-053e7c47012341234", "subnet-027cfea4b12341234" ] }, "RoleArn": "arn:aws:iam::account-id:role/MyEKSClusterRole" }, "End": true } } } ``` The following includes a `Task` state that deletes an Amazon EKS cluster. ``` { "StartAt": "DeleteCluster.sync", "States": { "DeleteCluster.sync": { "Type": "Task", "Resource": "arn:aws:states:::eks:deleteCluster.sync", "Arguments": { "Name": "MyCluster" }, "End": true } } } ``` The following includes a `Task` state that creates a Fargate profile. ``` { "StartAt": "CreateFargateProfile.sync", "States": { "CreateFargateProfile.sync": { "Type": "Task", "Resource": "arn:aws:states:::eks:createFargateProfile.sync", "Arguments": { "ClusterName": "MyCluster", "FargateProfileName": "MyFargateProfile", "PodExecutionRoleArn": "arn:aws:iam::account-id:role/MyFargatePodExecutionRole", "Selectors": [{ "Namespace": "my-namespace", "Labels": { "my-label": "my-value" } }] }, "End": true } } } ``` The following includes a `Task` state that deletes a Fargate profile. ``` { "StartAt": "DeleteFargateProfile.sync", "States": { "DeleteFargateProfile.sync": { "Type": "Task", "Resource": "arn:aws:states:::eks:deleteFargateProfile.sync", "Arguments": { "ClusterName": "MyCluster", "FargateProfileName": "MyFargateProfile" }, "End": true } } } ``` The following includes a `Task` state that creates a node group. ``` { "StartAt": "CreateNodegroup.sync", "States": { "CreateNodegroup.sync": { "Type": "Task", "Resource": "arn:aws:states:::eks:createNodegroup.sync", "Arguments": { "ClusterName": "MyCluster", "NodegroupName": "MyNodegroup", "NodeRole": "arn:aws:iam::account-id:role/MyNodeInstanceRole", "Subnets": ["subnet-09fb51df01234", "subnet-027cfea4b1234"] }, "End": true } } } ``` The following includes a `Task` state that deletes a node group. ``` { "StartAt": "DeleteNodegroup.sync", "States": { "DeleteNodegroup.sync": { "Type": "Task", "Resource": "arn:aws:states:::eks:deleteNodegroup.sync", "Arguments": { "ClusterName": "MyCluster", "NodegroupName": "MyNodegroup" }, "End": true } } } ``` ## Permissions When an Amazon EKS cluster is created using the `eks:createCluster` service integration, the IAM role is added to the Kubernetes RBAC authorization table as the administrator, with `system:masters` permissions. Initially, only that IAM entity can make calls to the Kubernetes API server. For example, you will not be able to use **kubectl** to interact with your Kubernetes API server, unless you assume the same role as your Step Functions state machine, or if you configure Kubernetes to grant permissions to additional IAM entities. For more information, see [Managing users or IAM roles for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) in the *Amazon EKS User Guide*. You can add permission for additional IAM entities, such as users or roles, by adding them to the `aws-auth` `ConfigMap` in the kube-system namespace. If you are creating your cluster from Step Functions, use the `eks:call` service integration. The following includes a `Task` state that creates an `aws-auth` `ConfigMap` and grants `system:masters` permission to the user `arn:aws:iam::account-id:user/my-user` and the IAM role `arn:aws:iam::account-id:role/my-role`. ``` { "StartAt": "Add authorized user", "States": { "Add authorized user": { "Type": "Task", "Resource": "arn:aws:states:::eks:call", "Arguments": { "ClusterName": "MyCluster", "CertificateAuthority": "LS0tLS1CRUd...UtLS0tLQo=", "Endpoint": "https://444455556666.yl4.region.eks.amazonaws.com", "Method": "POST", "Path": "/api/v1/namespaces/kube-system/configmaps", "RequestBody": { "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "aws-auth", "namespace": "kube-system" }, "data": { "mapUsers": "[{ \"userarn\": \"arn:aws:iam::account-id:user/my-user\", \"username\": \"my-user\", \"groups\": [ \"system:masters\" ] } ]", "mapRoles": "[{ \"rolearn\": \"arn:aws:iam::account-id:role/my-role\", \"username\": \"my-role\", \"groups\": [ \"system:masters\" ] } ]" } } }, "End": true } } ``` **Note** You may see the ARN for an IAM role displayed in a format that includes the path **/service-role/**, such as `arn:aws:iam::account-id:role/service-role/my-role`. This **service-role** path token should not be included when listing the role in `aws-auth`. When your cluster is first created the `aws-auth` `ConfigMap` will not exist, but will be added automatically if you create a Fargate profile. You can retrieve the current value of `aws-auth`, add the additional permissions, and `PUT` a new version. It is usually easier to create `aws-auth` before the Fargate profile. If your cluster was created outside of Step Functions, you can configure **kubectl** to communicate with your Kubernetes API server. Then, create a new `aws-auth` `ConfigMap` using `kubectl apply -f aws-auth.yaml` or edit one that already exists using `kubectl edit -n kube-system configmap/aws-auth`. For more information, see: + [Create a kubeconfig for Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html) in the *Amazon EKS User Guide*. + [Managing users or IAM roles for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) in the *Amazon EKS User Guide*. If your IAM role does not have sufficient permissions in Kubernetes, the `eks:call` or `eks:runJob` service integrations will fail with the following error: ``` Error: EKS.401 Cause: { "ResponseBody": { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 }, "StatusCode": 401, "StatusText": "Unauthorized" } ``` ## IAM policies for calling Amazon EKS The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). ### `CreateCluster` *Resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "eks:CreateCluster" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "eks:DescribeCluster", "eks:DeleteCluster" ], "Resource": "arn:aws:eks:us-east-1:444455556666:cluster/*" }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::444455556666:role/StepFunctionsSample-EKSClusterManag-EKSServiceRole-ANPAJ2UCCR6DPCEXAMPLE" ], "Condition": { "StringEquals": { "iam:PassedToService": "eks.amazonaws.com" } } } ] } ``` ``` ``` ### `CreateNodeGroup` *Resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeSubnets", "eks:CreateNodegroup" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "eks:DescribeNodegroup", "eks:DeleteNodegroup" ], "Resource": "arn:aws:eks:us-east-1:444455556666:nodegroup/*" }, { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:ListAttachedRolePolicies" ], "Resource": "arn:aws:iam::444455556666:role/*" }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::444455556666:role/StepFunctionsSample-EKSClusterMan-NodeInstanceRole-ANPAJ2UCCR6DPCEXAMPLE" ], "Condition": { "StringEquals": { "iam:PassedToService": "eks.amazonaws.com" } } } ] } ``` ### `DeleteCluster` *Resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "eks:DeleteCluster", "eks:DescribeCluster" ], "Resource": [ "arn:aws:eks:us-east-1:444455556666:cluster/ExampleCluster" ] } ] } ``` ### `DeleteNodegroup` *Resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "eks:DeleteNodegroup", "eks:DescribeNodegroup" ], "Resource": [ "arn:aws:eks:us-east-1:444455556666:nodegroup/ExampleCluster/ExampleNodegroup/*" ] } ] } ``` For more information about using Amazon EKS with Step Functions, see [Create and manage Amazon EKS clusters with Step Functions](#connect-eks). # Create and manage Amazon EMR clusters with Step Functions Learn how to integrate AWS Step Functions with Amazon EMR using the provided Amazon EMR service integration APIs. The service integration APIs are similar to the corresponding Amazon EMR APIs, with some differences in the fields that are passed and in the responses that are returned. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized Amazon EMR integration** The Optimized Amazon EMR service integration has a customized set of APIs that wrap the underlying Amazon EMR APIs, described below. Because of this, it differs significantly from the Amazon EMR AWS SDK service integration. The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. Step Functions does not terminate an Amazon EMR cluster automatically if execution is stopped. If your state machine stops before your Amazon EMR cluster has terminated, your cluster may continue running indefinitely, and can accrue additional charges. To avoid this, ensure that any Amazon EMR cluster you create is terminated properly. For more information, see: + [Control Cluster Termination](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-termination.html) in the Amazon EMR User Guide. + The Service Integration Patterns [Run a Job (.sync)](connect-to-resource.md#connect-sync) section. **Note** As of `emr-5.28.0`, you can specify the parameter `StepConcurrencyLevel` when creating a cluster to allow multiple steps to run in parallel on a single cluster. You can use the Step Functions `Map` and `Parallel` states to submit work in parallel to the cluster. The availability of Amazon EMR service integration is subject to the availability of Amazon EMR APIs. See [Amazon EMR](https://docs.aws.amazon.com//govcloud-us/latest/UserGuide/govcloud-emr.html) documentation for limitations in special regions. **Note** For integration with Amazon EMR, Step Functions has a hard-coded 60 seconds job polling frequency for the first 10 minutes and 300 seconds after that. ## Optimized Amazon EMR APIs The following table describes the differences between each Amazon EMR service integration API and corresponding Amazon EMR APIs. | Amazon EMR Service Integration API | Corresponding EMR API | Differences | | --- | --- | --- | | createCluster Creates and starts running a cluster (job flow). Amazon EMR is linked directly to a unique type of IAM role known as a service-linked role. For `createCluster` and `createCluster.sync` to work, you must have configured the necessary permissions to create the service-linked role `AWSServiceRoleForEMRCleanup`. For more information about this, including a statement you can add to your IAM permissions policy, see [Using the Service-Linked Role for Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/using-service-linked-roles.html). | [runJobFlow](https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) | createCluster uses the same request syntax as [runJobFlow](https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html), except for the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html)Response is this: 
{
  "ClusterId": "string"
}
Amazon EMR uses this: 
{
  "JobFlowId": "string"
}
| | createCluster.sync Creates and starts running a cluster (job flow). | [runJobFlow](https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) | The same as createCluster, but waits for the cluster to reach the WAITING state. | | setClusterTerminationProtection Locks a cluster (job flow) so the EC2 instances in the cluster cannot be terminated by user intervention, an API call, or a job-flow error. | [setTerminationProtection](https://docs.aws.amazon.com/emr/latest/APIReference/API_SetTerminationProtection.html) | Request uses this:
{
  "ClusterId": "string"
}
Amazon EMR uses this:
{
  "JobFlowIds": ["string"]
}
| | terminateCluster Shuts down a cluster (job flow). | [terminateJobFlows](https://docs.aws.amazon.com/emr/latest/APIReference/API_TerminateJobFlows.html) | Request uses this:
{
  "ClusterId": "string"
}
Amazon EMR uses this:
{
  "JobFlowIds": ["string"]
}
| | terminateCluster.syncShuts down a cluster (job flow). | [terminateJobFlows](https://docs.aws.amazon.com/emr/latest/APIReference/API_TerminateJobFlows.html) | The same as terminateCluster, but waits for the cluster to terminate. | | addStep Adds a new step to a running cluster. Optionally, you can also specify the `[ExecutionRoleArn](https://docs.aws.amazon.com/emr/latest/APIReference/API_AddJobFlowSteps.html#EMR-AddJobFlowSteps-request-ExecutionRoleArn)` parameter while using this API. | [addJobFlowSteps](https://docs.aws.amazon.com/emr/latest/APIReference/API_AddJobFlowSteps.html) | Request uses the key "ClusterId". Amazon EMR uses "JobFlowId". Request uses a single step.
{
  "Step": <"StepConfig object">
}
Amazon EMR uses this:
{
  "Steps": []
}
Response is this:
{
  "StepId": "string"
}
Amazon EMR returns this:
{
  "StepIds": []
}
| | addStep.sync Adds a new step to a running cluster. Optionally, you can also specify the `[ExecutionRoleArn](https://docs.aws.amazon.com/emr/latest/APIReference/API_AddJobFlowSteps.html#EMR-AddJobFlowSteps-request-ExecutionRoleArn)` parameter while using this API. | [addJobFlowSteps](https://docs.aws.amazon.com/emr/latest/APIReference/API_AddJobFlowSteps.html) | The same as addStep, but waits for the step to complete. | | cancelStep Cancels a pending step in a running cluster. | [cancelSteps](https://docs.aws.amazon.com/emr/latest/APIReference/API_CancelSteps.html) | Request uses this:
{
  "StepId": "string"
}
Amazon EMR uses this:
{
  "StepIds": []
}
Response is this:
{
  "CancelStepsInfo": 
}
Amazon EMR uses this:
{
  "CancelStepsInfoList": []
}
| | modifyInstanceFleetByName Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified `InstanceFleetName`. | [modifyInstanceFleet](https://docs.aws.amazon.com/emr/latest/APIReference/API_ModifyInstanceFleet.html) | Request is the same as for modifyInstanceFleet, except for the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html) | | modifyInstanceGroupByName Modifies the number of nodes and configuration settings of an instance group. | [modifyInstanceGroups](https://docs.aws.amazon.com/emr/latest/APIReference/API_ModifyInstanceGroups.html) | Request is this:
{
  "ClusterId": "string",
  "InstanceGroup": 
}
Amazon EMR uses a list: 
{
  "ClusterId": ["string"],
  "InstanceGroups": []
}
Within the `InstanceGroupModifyConfig` object, the field `InstanceGroupId` is not allowed. A new field, `InstanceGroupName`, has been added. At runtime the `InstanceGroupId` is determined automatically by the service integration by calling `ListInstanceGroups` and parsing the result. | ## Workflow example The following includes a `Task` state that creates a cluster. ``` "Create_Cluster": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync", "Arguments": { "Name": "MyWorkflowCluster", "VisibleToAllUsers": true, "ReleaseLabel": "emr-5.28.0", "Applications": [ { "Name": "Hive" } ], "ServiceRole": "EMR_DefaultRole", "JobFlowRole": "EMR_EC2_DefaultRole", "LogUri": "s3n://aws-logs-account-id-us-east-1/elasticmapreduce/", "Instances": { "KeepJobFlowAliveWhenNoSteps": true, "InstanceFleets": [ { "InstanceFleetType": "MASTER", "Name": "MASTER", "TargetOnDemandCapacity": 1, "InstanceTypeConfigs": [ { "InstanceType": "m4.xlarge" } ] }, { "InstanceFleetType": "CORE", "Name": "CORE", "TargetOnDemandCapacity": 1, "InstanceTypeConfigs": [ { "InstanceType": "m4.xlarge" } ] } ] } }, "End": true } ``` The following includes a `Task` state that enables termination protection. ``` "Enable_Termination_Protection": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:setClusterTerminationProtection", "Arguments": { "ClusterId": "{% $ClusterId %}", "TerminationProtected": true }, "End": true } ``` The following includes a `Task` state that submits a step to a cluster. ``` "Step_One": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync", "Arguments": { "ClusterId": "{% $ClusterId %}", "ExecutionRoleArn": "arn:aws:iam::account-id:role/myEMR-execution-role", "Step": { "Name": "The first step", "ActionOnFailure": "TERMINATE_CLUSTER", "HadoopJarStep": { "Jar": "command-runner.jar", "Args": [ "hive-script", "--run-hive-script", "--args", "-f", "s3://region.elasticmapreduce.samples/cloudfront/code/Hive_CloudFront.q", "-d", "INPUT=s3://region.elasticmapreduce.samples", "-d", "OUTPUT=s3:///MyHiveQueryResults/" ] } } }, "End": true } ``` The following includes a `Task` state that cancels a step. ``` "Cancel_Step_One": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:cancelStep", "Arguments": { "ClusterId": "{% $ClusterId %}", "StepId": "{% $AddStepsResult.StepId %}" }, "End": true } ``` The following includes a `Task` state that terminates a cluster. ``` "Terminate_Cluster": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:terminateCluster.sync", "Arguments": { "ClusterId": "{% $ClusterId %}", }, "End": true } ``` The following includes a `Task` state that scales a cluster up or down for an instance group. ``` "ModifyInstanceGroupByName": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:modifyInstanceGroupByName", "Arguments": { "ClusterId": "j-account-id3", "InstanceGroupName": "MyCoreGroup", "InstanceGroup": { "InstanceCount": 8 } }, "End": true } ``` The following includes a `Task` state that scales a cluster up or down for an instance fleet. ``` "ModifyInstanceFleetByName": { "Type": "Task", "Resource": "arn:aws:states:::elasticmapreduce:modifyInstanceFleetByName", "Arguments": { "ClusterId": "j-account-id3", "InstanceFleetName": "MyCoreFleet", "InstanceFleet": { "TargetOnDemandCapacity": 8, "TargetSpotCapacity": 0 } }, "End": true } ``` ## IAM policies for calling Amazon EMR The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). ### `addStep` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:AddJobFlowSteps", "elasticmapreduce:DescribeStep", "elasticmapreduce:CancelSteps" ], "Resource": [ "arn:aws:elasticmapreduce:us-east-1:123456789012:cluster/clusterId" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:AddJobFlowSteps", "elasticmapreduce:DescribeStep", "elasticmapreduce:CancelSteps" ], "Resource": "arn:aws:elasticmapreduce:*:*:cluster/*" } ] } ``` ### `cancelStep` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "elasticmapreduce:CancelSteps", "Resource": [ "arn:aws:elasticmapreduce:us-east-1:123456789012:cluster/myCluster-id" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "elasticmapreduce:CancelSteps", "Resource": "arn:aws:elasticmapreduce:*:*:cluster/*" } ] } ``` ### `createCluster` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:RunJobFlow", "elasticmapreduce:DescribeCluster", "elasticmapreduce:TerminateJobFlows" ], "Resource": "*" }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::123456789012:role/myRoleName" ] } ] } ``` ### `setClusterTerminationProtection` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "elasticmapreduce:SetTerminationProtection", "Resource": [ "arn:aws:elasticmapreduce:us-east-1:123456789012:cluster/myCluster-id" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "elasticmapreduce:SetTerminationProtection", "Resource": "arn:aws:elasticmapreduce:*:*:cluster/*" } ] } ``` ### `modifyInstanceFleetByName` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:ModifyInstanceFleet", "elasticmapreduce:ListInstanceFleets" ], "Resource": [ "arn:aws:elasticmapreduce:us-east-1:123456789012:cluster/myCluster-id" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:ModifyInstanceFleet", "elasticmapreduce:ListInstanceFleets" ], "Resource": "arn:aws:elasticmapreduce:*:*:cluster/*" } ] } ``` ### `modifyInstanceGroupByName` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:ModifyInstanceGroups", "elasticmapreduce:ListInstanceGroups" ], "Resource": [ "arn:aws:elasticmapreduce:us-east-1:123456789012:cluster/myCluster-id" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:ModifyInstanceGroups", "elasticmapreduce:ListInstanceGroups" ], "Resource": "*" } ] } ``` ### `terminateCluster` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:TerminateJobFlows", "elasticmapreduce:DescribeCluster" ], "Resource": [ "arn:aws:elasticmapreduce:us-east-1:123456789012:cluster/myCluster-id" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticmapreduce:TerminateJobFlows", "elasticmapreduce:DescribeCluster" ], "Resource": "arn:aws:elasticmapreduce:*:*:cluster/*" } ] } ``` # Create and manage Amazon EMR clusters on EKS with AWS Step Functions Learn how to integrate AWS Step Functions with Amazon EMR on EKS using the Amazon EMR on EKS service integration APIs. The service integration APIs are the same as the corresponding Amazon EMR on EKS APIs, but not all APIs support all integration patterns, as shown in the following table. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **How the Optimized Amazon EMR on EKS integration is different than the Amazon EMR on EKS AWS SDK integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. There are no specific optimizations for the [Request Response](connect-to-resource.md#connect-default) integration pattern. The [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration pattern is not supported. **Note** For integration with Amazon EMR, Step Functions has a hard-coded 60 seconds job polling frequency for the first 10 minutes and 300 seconds after that. | API | Request response | Run a job (.sync) | | --- | --- | --- | | CreateVirtualCluster | Supported | Not supported | | DeleteVirtualCluster | Supported | Supported | | StartJobRun | Supported | Supported | Supported Amazon EMR on EKS APIs: **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). + [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html) + [Request syntax](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html#API_CreateVirtualCluster_RequestSyntax) + [Supported parameters](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html#API_CreateVirtualCluster_RequestBody) + [Response syntax](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html#API_CreateVirtualCluster_ResponseSyntax) + [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html) + [Request syntax](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html#API_DeleteVirtualCluster_RequestSyntax) + [Supported parameters](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_DeleteVirtualCluster.html#API_DeleteVirtualCluster_RequestParameters) + [Response syntax](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_CreateVirtualCluster.html#API_CreateVirtualCluster_ResponseSyntax) + [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) + [Request syntax](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html#API_StartJobRun_RequestSyntax) + [Supported parameters](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html#API_StartJobRun_RequestParameters) + [Response syntax](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html#API_StartJobRun_ResponseSyntax) The following includes a `Task` state that creates a virtual cluster. ``` "Create_Virtual_Cluster": { "Type": "Task", "Resource": "arn:aws:states:::emr-containers:createVirtualCluster", "Arguments": { "Name": "MyVirtualCluster", "ContainerProvider": { "Id": "EKSClusterName", "Type": "EKS", "Info": { "EksInfo": { "Namespace": "Namespace" } } } }, "End": true } ``` The following includes a `Task` state that submits a job to a virtual cluster and waits for it to complete. ``` "Submit_Job": { "Type": "Task", "Resource": "arn:aws:states:::emr-containers:startJobRun.sync", "Arguments": { "Name": "MyJobName", "VirtualClusterId": "{% $VirtualClusterId %}", "ExecutionRoleArn": "arn:aws:iam:::role/job-execution-role", "ReleaseLabel": "emr-6.2.0-latest", "JobDriver": { "SparkSubmitJobDriver": { "EntryPoint": "s3:///jobs/trip-count.py", "EntryPointArguments": [ "60" ], "SparkSubmitParameters": "--conf spark.driver.cores=2 --conf spark.executor.instances=10 --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.executor.memory=10G --conf spark.driver.memory=10G --conf spark.executor.cores=1 --conf spark.dynamicAllocation.enabled=false" } }, "ConfigurationOverrides": { "ApplicationConfiguration": [ { "Classification": "spark-defaults", "Properties": { "spark.executor.instances": "2", "spark.executor.memory": "2G" } } ], "MonitoringConfiguration": { "PersistentAppUI": "ENABLED", "CloudWatchMonitoringConfiguration": { "LogGroupName": "MyLogGroupName", "LogStreamNamePrefix": "MyLogStreamNamePrefix" }, "S3MonitoringConfiguration": { "LogUri": "s3://" } } }, "Tags": { "taskType": "jobName" } }, "End": true } ``` The following includes a `Task` state that deletes a virtual cluster and waits for the deletion to complete. ``` "Delete_Virtual_Cluster": { "Type": "Task", "Resource": "arn:aws:states:::emr-containers:deleteVirtualCluster.sync", "Arguments": { "Id": "{% $states.input.VirtualClusterId %}", }, "End": true } ``` To learn about configuring IAM permissions when using Step Functions with other AWS services, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md). # Create and manage Amazon EMR Serverless applications with Step Functions Learn how to create, start, stop, and delete applications on EMR Serverless using Step Functions. This page lists the supported APIs and provides example `Task` states to perform common use cases. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized EMR Serverless integration** The Optimized EMR Serverless service integration has a customized set of [APIs](#connect-emr-serverless-custom-apis) that wrap the underlying EMR Serverless APIs. Because of this customization, the optimized EMR Serverless integration differs significantly from the AWS SDK service integration. In addition, the optimized EMR Serverless integration supports [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern. The [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration pattern is **not** supported. ## EMR Serverless service integration APIs To integrate AWS Step Functions with EMR Serverless, you can use the following six EMR Serverless service integration APIs. These service integration APIs are similar to the corresponding EMR Serverless APIs, with some differences in the fields that are passed and in the responses that are returned. The following table describes the differences between each EMR Serverless service integration API and its corresponding EMR Serverless API. | EMR Serverless service integration API | Corresponding EMR Serverless API | Differences | | --- | --- | --- | | *createApplication* Creates an application. EMR Serverless is linked to a unique type of IAM role known as a service-linked role. For `createApplication` and `createApplication.sync` to work, you must have configured the necessary permissions to create the service-linked role `AWSServiceRoleForAmazonEMRServerless`. For more information about this, including a statement you can add to your IAM permissions policy, see [Using service-linked roles for EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/using-service-linked-roles.html). | [CreateApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_CreateApplication.html) | None | | *createApplication.sync* Creates an application. | [CreateApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_CreateApplication.html) | No differences between the requests and responses of the EMR Serverless API and EMR Serverless service integration API. However, *createApplication.sync* waits for the application to reach the `CREATED` state. | | *startApplication* Starts a specified application and initializes the application's initial capacity if configured. | [StartApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartApplication.html) | The EMR Serverless API response doesn't contain any data, but the EMR Serverless service integration API response includes the following data. 
{
  "ApplicationId": "string"
}
| | *startApplication.sync* Starts a specified application and initializes the initial capacity if configured. | [StartApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartApplication.html) | The EMR Serverless API response doesn't contain any data, but the EMR Serverless service integration API response includes the following data. 
{
  "ApplicationId": "string"
}
Also, *startApplication.sync* waits for the application to reach the `STARTED` state. | | *stopApplication* Stops a specified application and releases initial capacity if configured. All scheduled and running jobs must be completed or cancelled before stopping an application. | [StopApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StopApplication.html) | The EMR Serverless API response doesn't contain any data, but the EMR Serverless service integration API response includes the following data. 
{
  "ApplicationId": "string"
}
| | *stopApplication.sync* Stops a specified application and releases initial capacity if configured. All scheduled and running jobs must be completed or cancelled before stopping an application. | [StopApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StopApplication.html) | The EMR Serverless API response doesn't contain any data, but the EMR Serverless service integration API response includes the following data. 
{
  "ApplicationId": "string"
}
Also, *stopApplication.sync* waits for the application to reach the `STOPPED` state. | | *deleteApplication* Deletes an application. An application must be in the `STOPPED` or `CREATED` state in order to be deleted. | [DeleteApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_DeleteApplication.html) | The EMR Serverless API response doesn't contain any data, but the EMR Serverless service integration API response includes the following data. 
{
  "ApplicationId": "string"
}
| | *deleteApplication.sync* Deletes an application. An application must be in the `STOPPED` or `CREATED` state in order to be deleted. | [DeleteApplication](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_DeleteApplication.html) | The EMR Serverless API response doesn't contain any data, but the EMR Serverless service integration API response includes the following data. 
{
  "ApplicationId": "string"
}
Also, *stopApplication.sync* waits for the application to reach the `TERMINATED` state. | | *startJobRun* Starts a job run. | [StartJobRun](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html) | None | | *startJobRun.sync* Starts a job run. | [StartJobRun](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html) | No differences between the requests and responses of the EMR Serverless API and EMR Serverless service integration API. However, *startJobRun.sync* waits for the application to reach the `SUCCESS` state. | | *cancelJobRun* Cancels a job run. | [CancelJobRun](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_CancelJobRun.html) | None | | *cancelJobRun.sync* Cancels a job run. | [CancelJobRun](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_CancelJobRun.html) | No differences between the requests and responses of the EMR Serverless API and EMR Serverless service integration API. However, *cancelJobRun.sync* waits for the application to reach the `CANCELLED` state. | ## EMR Serverless integration use cases For the Optimized EMR Serverless service integration, we recommend that you create a single application, and then use that application to run multiple jobs. For example, in a single state machine, you can include multiple [startJobRun](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html) requests, all of which use the same application. The following [Task workflow state](state-task.md) state examples show use cases to integrate EMR Serverless APIs with Step Functions. For information about other use cases of EMR Serverless, see [What is Amazon EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html). **Tip** To deploy an example of a state machine that integrates with EMR Serverless for running multiple jobs;, see [Run an EMR Serverless job](sample-emr-serverless-job.md). + [Create an application](#connect-emr-serverless-task-state-createapp) + [Start an application](#connect-emr-serverless-task-state-startapp) + [Stop an application](#connect-emr-serverless-task-state-stopapp) + [Delete an application](#connect-emr-serverless-task-state-deleteapp) + [Start a job in an application](#connect-emr-serverless-task-state-startjobrun) + [Cancel a job in an application](#connect-emr-serverless-task-state-canceljobrun) To learn about configuring IAM permissions when using Step Functions with other AWS services, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md). In the examples shown in the following use cases, replace the *italicized* text with your resource-specific information. For example, replace *yourApplicationId* with the ID of your EMR Serverless application, such as `00yv7iv71inak893`. ### Create an application The following Task state example creates an application using the *createApplication.sync* service integration API. ``` "Create_Application": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:createApplication.sync", "Arguments": { "Name": "MyApplication", "ReleaseLabel": "emr-6.9.0", "Type": "SPARK" }, "End": true } ``` ### Start an application The following Task state example starts an application using the *startApplication.sync* service integration API. ``` "Start_Application": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:startApplication.sync", "Arguments": { "ApplicationId": "yourApplicationId" }, "End": true } ``` ### Stop an application The following Task state example stops an application using the *stopApplication.sync* service integration API. ``` "Stop_Application": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:stopApplication.sync", "Arguments": { "ApplicationId": "yourApplicationId" }, "End": true } ``` ### Delete an application The following Task state example deletes an application using the *deleteApplication.sync* service integration API. ``` "Delete_Application": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:deleteApplication.sync", "Arguments": { "ApplicationId": "yourApplicationId" }, "End": true } ``` ### Start a job in an application The following Task state example starts a job in an application using the *startJobRun.sync* service integration API. ``` "Start_Job": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:startJobRun.sync", "Arguments": { "ApplicationId": "yourApplicationId", "ExecutionRoleArn": "arn:aws:iam::account-id:role/myEMRServerless-execution-role", "JobDriver": { "SparkSubmit": { "EntryPoint": "s3:///sample.py", "EntryPointArguments": ["1"], "SparkSubmitParameters": "--conf spark.executor.cores=4 --conf spark.executor.memory=4g --conf spark.driver.cores=2 --conf spark.driver.memory=4g --conf spark.executor.instances=1" } } }, "End": true } ``` ### Cancel a job in an application The following Task state example cancels a job in an application using the *cancelJobRun.sync* service integration API. ``` "Cancel_Job": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:cancelJobRun.sync", "Arguments": { "ApplicationId": "{% $states.input.ApplicationId %}", "JobRunId": "{% $states.input.JobRunId %}" }, "End": true } ``` ## IAM policies for calling Amazon EMR Serverless When you create a state machine using the console, Step Functions automatically creates an execution role for your state machine with the least privileges required. These automatically generated IAM roles are valid for the AWS Region in which you create the state machine. The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). We recommend that when you create IAM policies, do not include wildcards in the policies. As a security best practice, you should scope your policies down as much as possible. You should use dynamic policies only when certain input parameters are not known during runtime. Further, administrator users should be careful when granting non-administrator users execution roles for running the state machines. We recommend that you include passRole policies in the execution roles if you're creating policies on your own. We also recommend that you add the `aws:SourceARN` and `aws:SourceAccount` context keys in the execution roles. ### IAM policy examples for EMR Serverless integration with Step Functions + [IAM policy example for CreateApplication](#emr-serverless-policy-createapp) + [IAM policy example for StartApplication](#emr-serverless-policy-startapp) + [IAM policy example for StopApplication](#emr-serverless-policy-stopapp) + [IAM policy example for DeleteApplication](#emr-serverless-policy-deleteapp) + [IAM policy example for StartJobRun](#emr-serverless-policy-startjobrun) + [IAM policy example for CancelJobRun](#emr-serverless-policy-canceljobrun) #### IAM policy example for CreateApplication The following is an IAM policy example for a state machine with a CreateApplication [Task workflow state](state-task.md) state. **Note** You need to specify the CreateServiceLinkedRole permissions in your IAM policies during the creation of the first ever application in your account. Thereafter, you need not add this permission. For information about CreateServiceLinkedRole, see [CreateServiceLinkedRole](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateServiceLinkedRole.html) in the https://docs.aws.amazon.com/IAM/latest/APIReference/. Static and dynamic resources for the following policies are the same. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:CreateApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/*" ] }, { "Effect": "Allow", "Action": [ "emr-serverless:GetApplication", "emr-serverless:DeleteApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "arn:aws:iam::123456789012:role/aws-service-role/ops.emr-serverless.amazonaws.com/AWSServiceRoleForAmazonEMRServerless*", "Condition": { "StringLike": { "iam:AWSServiceName": "ops.emr-serverless.amazonaws.com" } } } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:CreateApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/*" ] }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "arn:aws:iam::123456789012:role/aws-service-role/ops.emr-serverless.amazonaws.com/AWSServiceRoleForAmazonEMRServerless*", "Condition": { "StringLike": { "iam:AWSServiceName": "ops.emr-serverless.amazonaws.com" } } } ] } ``` ------ #### IAM policy example for StartApplication **Static resources** The following are IAM policy examples for static resources when you use a state machine with a StartApplication [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartApplication", "emr-serverless:GetApplication", "emr-serverless:StopApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] } ] } ``` ------ **Dynamic resources** The following are IAM policy examples for dynamic resources when you use a state machine with a StartApplication [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartApplication", "emr-serverless:GetApplication", "emr-serverless:StopApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] } ] } ``` ------ #### IAM policy example for StopApplication **Static resources** The following are IAM policy examples for static resources when you use a state machine with a StopApplication [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StopApplication", "emr-serverless:GetApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StopApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] } ] } ``` ------ **Dynamic resources** The following are IAM policy examples for dynamic resources when you use a state machine with a StopApplication [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StopApplication", "emr-serverless:GetApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StopApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] } ] } ``` ------ #### IAM policy example for DeleteApplication **Static resources** The following are IAM policy examples for static resources when you use a state machine with a DeleteApplication [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:DeleteApplication", "emr-serverless:GetApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:DeleteApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] } ] } ``` ------ **Dynamic resources** The following are IAM policy examples for dynamic resources when you use a state machine with a DeleteApplication [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:DeleteApplication", "emr-serverless:GetApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessApplicationRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:DeleteApplication" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] } ] } ``` ------ #### IAM policy example for StartJobRun **Static resources** The following are IAM policy examples for static resources when you use a state machine with a StartJobRun [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::123456789012:role/jobExecutionRoleArn" ], "Condition": { "StringEquals": { "iam:PassedToService": "emr-serverless.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "emr-serverless:GetJobRun", "emr-serverless:CancelJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId/jobruns/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessJobRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId" ] }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::123456789012:role/jobExecutionRoleArn" ], "Condition": { "StringEquals": { "iam:PassedToService": "emr-serverless.amazonaws.com" } } } ] } ``` ------ **Dynamic resources** The following are IAM policy examples for dynamic resources when you use a state machine with a StartJobRun [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartJobRun", "emr-serverless:GetJobRun", "emr-serverless:CancelJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::123456789012:role/jobExecutionRoleArn" ], "Condition": { "StringEquals": { "iam:PassedToService": "emr-serverless.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessJobRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:StartJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::123456789012:role/jobExecutionRoleArn" ], "Condition": { "StringEquals": { "iam:PassedToService": "emr-serverless.amazonaws.com" } } } ] } ``` ------ #### IAM policy example for CancelJobRun **Static resources** The following are IAM policy examples for static resources when you use a state machine with a CancelJobRun [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:CancelJobRun", "emr-serverless:GetJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId/jobruns/jobRunId" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessJobRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:CancelJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/applicationId/jobruns/jobRunId" ] } ] } ``` ------ **Dynamic resources** The following are IAM policy examples for dynamic resources when you use a state machine with a CancelJobRun [Task workflow state](state-task.md) state. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:CancelJobRun", "emr-serverless:GetJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForEMRServerlessJobRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "emr-serverless:CancelJobRun" ], "Resource": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/*" ] } ] } ``` ------ # Add EventBridge events with Step Functions Step Functions provides a service integration API for integrating with Amazon EventBridge. Learn how to build event-driven applications by sending custom events directly from Step Functions workflows. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized EventBridge integration** The execution ARN and the state machine ARN are automatically appended to the `Resources` field of each `PutEventsRequestEntry`. If the response from `PutEvents` contains a non-zero `FailedEntryCount` then the `Task` state fails with the error `EventBridge.FailedEntry`. To use the `PutEvents` API, you will need to create an EventBridge rule in your account that matches the specific pattern of the events you will send. For example, you could: + Create a Lambda function in your account that receives and prints an event that matches an EventBridge rule. + Create an EventBridge rule in your account on the default event bus that matches a specific event pattern and targets the Lambda function. For more information, see: + [Adding Amazon EventBridge events with PutEvents](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html) in the EventBridge User Guide. + [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) in Service Integration Patterns. The following includes a `Task` that sends a custom event: ``` { "Type": "Task", "Resource": "arn:aws:states:::events:putEvents", "Arguments": { "Entries": [ { "Detail": { "Message": "MyMessage" }, "DetailType": "MyDetailType", "EventBusName": "MyEventBus", "Source": "my.source" } ] }, "End": true } ``` **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). ## Optimized EventBridge API Supported EventBridge API and syntax include: + [https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_PutEvents.html](https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_PutEvents.html) ## Error handling The `PutEvents` API accepts an array of entries as input, then returns an array of result entries. As long as the `PutEvents` action was successful, `PutEvents` will return an HTTP 200 response, even if one or more entries failed. `PutEvents` returns the number of failed entries in the `FailedEntryCount` field. Step Functions checks whether the `FailedEntryCount` is greater than zero. If it is greater than zero, Step Functions fails the state with the error `EventBridge.FailedEntry`. This lets you use the built-in error handling of Step Functions on task states to catch or retry when there are failed entries, rather than needing to use an additional state to analyze the `FailedEntryCount` from the response. **Note** If you have implemented idempotency and can safely retry on all entries, you can use Step Functions' retry logic. Step Functions does not remove successful entries from the `PutEvents` input array before retrying. Instead, it retries with the original array of entries. ## IAM policies for calling EventBridge The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). ### `PutEvents` *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "events:PutEvents" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:event-bus/my-project-eventbus" ], "Effect": "Allow" } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "events:PutEvents" ], "Resource": "arn:aws:events:*:*:event-bus/*" } ] } ``` # Start an AWS Glue job with Step Functions Learn to use Step Functions to start a job run on AWS Glue. This page lists the supported API actions and provides an example `Task` state to start a AWS Glue job. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized AWS Glue integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is available. The `JobName` field is extracted from the request and inserted into the response, which normally only contains `JobRunID`. The following includes a `Task` state that starts an AWS Glue job. ``` "Glue StartJobRun": { "Type": "Task", "Resource": "arn:aws:states:::glue:startJobRun.sync", "Arguments": { "JobName": "GlueJob-JTrRO5l98qMG" }, "Next": "ValidateOutput" }, ``` **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## Optimized AWS Glue APIs + [https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html#aws-glue-api-jobs-runs-StartJobRun](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html#aws-glue-api-jobs-runs-StartJobRun) ## IAM policies for calling AWS Glue The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). AWS Glue does not have resource-based control. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:StartJobRun", "glue:GetJobRun", "glue:GetJobRuns", "glue:BatchStopJobRun" ], "Resource": "*" } ] } ``` ------ #### [ Request Response and Callback (.waitForTaskToken) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:StartJobRun" ], "Resource": "*" } ] } ``` ------ # Start AWS Glue DataBrew jobs with Step Functions Learn how you can use the DataBrew integration to add data cleaning and data normalization steps into your analytics and machine learning workflows with Step Functions. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). The following includes a `Task` state that starts a request-response DataBrew job. ``` "DataBrew StartJobRun": { "Type": "Task", "Resource": "arn:aws:states:::databrew:startJobRun", "Arguments": { "Name": "sample-proj-job-1" }, "Next": "NEXT_STATE" }, ``` The following includes a `Task` state that starts a sync DataBrew job. ``` "DataBrew StartJobRun": { "Type": "Task", "Resource": "arn:aws:states:::databrew:startJobRun.sync", "Arguments": { "Name": "sample-proj-job-1" }, "Next": "NEXT_STATE" }, ``` **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## Supported DataBrew APIs + `[https://docs.aws.amazon.com/databrew/latest/dg/API_StartJobRun.html](https://docs.aws.amazon.com/databrew/latest/dg/API_StartJobRun.html)` ## IAM policies for calling DataBrew The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "databrew:startJobRun", "databrew:listJobRuns", "databrew:stopJobRun" ], "Resource": [ "arn:aws:databrew:us-east-1:123456789012:job/*" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "databrew:startJobRun" ], "Resource": [ "arn:aws:databrew:us-east-1:123456789012:job/*" ] } ] } ``` ------ # Invoke an AWS Lambda function with Step Functions Learn how to use Step Functions to invoke Lambda functions either synchronously or asynchronously as part of an event-driven serverless application. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized Lambda integration** The `Payload` field of the response is parsed from escaped Json to Json. If an exception is raised within the Lambda function, the Task will fail. For a practical example, see [Handling error conditions in a Step Functions state machine](tutorial-handling-error-conditions.md). ## Optimized Lambda APIs + [https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html](https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html) ## Workflow Examples The following includes a `Task` state that invokes a Lambda function. ``` { "StartAt":"CallLambda", "States":{ "CallLambda":{ "Type":"Task", "Resource":"arn:aws:states:::lambda:invoke", "Arguments":{ "FunctionName":"arn:aws:lambda:region:account-id:function:MyFunction" }, "End":true } } } ``` The following includes a `Task` state that implements the [callback](connect-to-resource.md#connect-wait-token) service integration pattern. ``` { "StartAt":"GetManualReview", "States":{ "GetManualReview":{ "Type":"Task", "Resource":"arn:aws:states:::lambda:invoke.waitForTaskToken", "Arguments":{ "FunctionName":"arn:aws:lambda:region:account-id:function:get-model-review-decision", "Payload":{ "model":"{% $states.input.my-model %}", "TaskToken": "{% $states.context.Task.Token %}" }, "Qualifier":"prod-v1" }, "End":true } } } ``` When you invoke a Lambda function, the execution will wait for the function to complete. If you invoke the Lambda function with a callback task, the heartbeat timeout does not start counting until after the Lambda function has completed executing and returned a result. As long as the Lambda function executes, the heartbeat timeout is not enforced. It is also possible to call Lambda asynchronously using the `InvocationType` parameter, as seen in the following example: ``` { "Comment": "A Hello World example of the Amazon States Language using Pass states", "StartAt": "Hello", "States": { "Hello": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Arguments": { "FunctionName": "arn:aws:lambda:region:account-id:function:echo", "InvocationType": "Event" }, "End": true } } } ``` **Note** For asynchronous invocations of Lambda functions, the heartbeat timeout period starts immediately. When the `Task` result is returned, the function output is nested inside a dictionary of metadata. For example: ``` { "ExecutedVersion":"$LATEST", "Payload":"FUNCTION OUTPUT", "SdkHttpMetadata":{ "HttpHeaders":{ "Connection":"keep-alive", "Content-Length":"4", "Content-Type":"application/json", "Date":"Fri, 26 Mar 2021 07:42:02 GMT", "X-Amz-Executed-Version":"$LATEST", "x-amzn-Remapped-Content-Length":"0", "x-amzn-RequestId":"0101aa0101-1111-111a-aa55-1010aaa1010", "X-Amzn-Trace-Id":"root=1-1a1a000a2a2-fe0101aa10ab;sampled=0" }, "HttpStatusCode":200 }, "SdkResponseMetadata":{ "RequestId":"6b3bebdb-9251-453a-ae45-512d9e2bf4d3" }, "StatusCode":200 } ``` ## Directly specified function resource Alternatively, you can invoke a Lambda function by specifying a function ARN directly in the "Resource" field. When you invoke a Lambda function in this way, you can't specify `.waitForTaskToken`, and the task result contains only the function output. ``` { "StartAt":"CallFunction", "States":{ "CallFunction": { "Type":"Task", "Resource":"arn:aws:lambda:region:account-id:function:HelloFunction", "End": true } } } ``` With this form of integration, the function could succeed yet send a response that contains a `FunctionError` field. In that scenario, the workflow Task will fail. You can invoke a specific Lambda function version or alias by specifying those options in the ARN in the `Resource` field. See the following in the Lambda documentation: + [AWS Lambda versioning](https://docs.aws.amazon.com/lambda/latest/dg/versioning-intro.html) + [AWS Lambda aliases](https://docs.aws.amazon.com/lambda/latest/dg/aliases-intro.html) ## IAM policies for calling AWS Lambda The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). In the following example, a state machine with two AWS Lambda task states which call `function1` and `function2`, the autogenerated policy includes `lambda:Invoke` permission for both functions. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": [ "arn:aws:lambda:us-east-1:123456789012:function:myFn1", "arn:aws:lambda:us-east-1:123456789012:function:myFn2" ] } ] } ``` # Create an AWS Elemental MediaConvert job with Step Functions Learn how to use Step Functions to create an AWS Elemental MediaConvert job using the [https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobspost](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobspost) API. **Experiment with Step Functions and MediaConvert** Learn how to use the MediaConvert optimized integration in a workflow that detects and removes SMTPE color bars of unknown length from the beginning of a video clip. Read the blog post from Apr, 12, 2024: [https://aws.amazon.com/blogs/media/low-code-workflows-with-aws-elemental-mediaconvert/](https://aws.amazon.com/blogs/media/low-code-workflows-with-aws-elemental-mediaconvert/) To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized MediaConvert integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) and [Request Response](connect-to-resource.md#connect-default) integration patterns are supported. Step Functions will add the following custom tag to MediaConvert jobs: `ManagedByService: AWSStepFunctions` There is no specific optimization for [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration patterns. The following includes a `Task` state that submits a MediaConvert job and waits for it to complete. ``` { "StartAt": "MediaConvert_CreateJob", "States": { "MediaConvert_CreateJob": { "Type": "Task", "Resource": "arn:aws:states:::mediaconvert:createJob.sync", "Arguments": { "Role": "arn:aws:iam::111122223333:role/Admin", "Settings": { "OutputGroups": [ { "Outputs": [ { "ContainerSettings": { "Container": "MP4" }, "VideoDescription": { "CodecSettings": { "Codec": "H_264", "H264Settings": { "MaxBitrate": 1000, "RateControlMode": "QVBR", "SceneChangeDetect": "TRANSITION_DETECTION" } } }, "AudioDescriptions": [ { "CodecSettings": { "Codec": "AAC", "AacSettings": { "Bitrate": 96000, "CodingMode": "CODING_MODE_2_0", "SampleRate": 48000 } } } ] } ], "OutputGroupSettings": { "Type": "FILE_GROUP_SETTINGS", "FileGroupSettings": { "Destination": "s3://amzn-s3-demo-destination-bucket/" } } } ], "Inputs": [ { "AudioSelectors": { "Audio Selector 1": { "DefaultSelection": "DEFAULT" } }, "FileInput": "s3://amzn-s3-demo-bucket/DOC-EXAMPLE-SOURCE_FILE" } ] } }, "End": true } } } ``` **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. ## Optimized MediaConvert APIs + [https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobspost](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobspost) + [Request syntax](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-request-body-post-example) + Supported parameters: + [https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-prop-createjobrequest-role](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-prop-createjobrequest-role) (Required) + [https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-prop-createjobrequest-settings](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-prop-createjobrequest-settings) (Required) + [https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-model-createjobrequest](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-model-createjobrequest) (Optional) + [Response syntax](https://docs.aws.amazon.com/mediaconvert/latest/apireference/jobs.html#jobs-response-examples) – see **CreateJobResponse schema** ## IAM policies for calling AWS Elemental MediaConvert The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). The IAM policy for `GetJob` and `CancelJob` actions are scoped to only permit access to jobs with the `ManagedByService: AWSStepFunctions` tag. **Tag-based policy** Modifying the autogenerated `ManagedByService: AWSStepFunctions` tag will cause state machine executions to fail. ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "MediaConvertCreateJob", "Effect": "Allow", "Action": [ "mediaconvert:CreateJob" ], "Resource": [ "arn:aws:mediaconvert:us-east-1:123456789012:queues/*", "arn:aws:mediaconvert:us-east-1:123456789012:jobTemplates/*", "arn:aws:mediaconvert:us-east-1:123456789012:presets/*" ] }, { "Sid": "MediaConvertManageJob", "Effect": "Allow", "Action": [ "mediaconvert:GetJob", "mediaconvert:CancelJob" ], "Resource": "arn:aws:mediaconvert:us-east-1:123456789012:jobs/*", "Condition": { "StringEquals": { "aws:ResourceTag/ManagedByService": "AWSStepFunctions" } } }, { "Sid": "IamPassRole", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRoleName" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "mediaconvert.amazonaws.com" ] } } }, { "Sid": "EventBridgeManageRule", "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForMediaConvertJobRule" ] } ] } ``` ------ #### [ Request Response ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "MediaConvertCreateJob", "Effect": "Allow", "Action": [ "mediaconvert:CreateJob" ], "Resource": [ "arn:aws:mediaconvert:us-east-1:123456789012:queues/*", "arn:aws:mediaconvert:us-east-1:123456789012:jobTemplates/*", "arn:aws:mediaconvert:us-east-1:123456789012:presets/*" ] }, { "Sid": "IamPassRole", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/myRoleName" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "mediaconvert.amazonaws.com" ] } } } ] } ``` ------ # Create and manage Amazon SageMaker AI jobs with Step Functions Learn how to use Step Functions to create and manage jobs on SageMaker AI. This page lists the supported SageMaker AI API actions and provides example `Task` states to create SageMaker AI transform, training, labeling, and processing jobs. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized SageMaker AI integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is supported. There are no specific optimizations for the [Request Response](connect-to-resource.md#connect-default) integration pattern. The [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration pattern is not supported. ## Optimized SageMaker AI APIs + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateEndpoint.html) + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateEndpointConfig.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateEndpointConfig.html) + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateHyperParameterTuningJob.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateHyperParameterTuningJob.html) - Supports the `.sync` integration pattern. + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateLabelingJob.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateLabelingJob.html) - Supports the `.sync` integration pattern. + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateModel.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateModel.html) + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateProcessingJob.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateProcessingJob.html) - Supports the `.sync` integration pattern. + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html) - Supports the `.sync` integration pattern. + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTransformJob.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTransformJob.html) - Supports the `.sync` integration pattern. + [https://docs.aws.amazon.com/sagemaker/latest/dg/API_UpdateEndpoint.html](https://docs.aws.amazon.com/sagemaker/latest/dg/API_UpdateEndpoint.html) **Note** AWS Step Functions will not automatically create a policy for `CreateTransformJob`. You must attach an inline policy to the created role. For more information, see this example IAM policy: [`CreateTrainingJob`](#sagemaker-iam-createtrainingjob). ## SageMaker AI Transform Job Example The following includes a `Task` state that creates an Amazon SageMaker AI transform job, specifying the Amazon S3 location for `DataSource` and `TransformOutput`. ``` { "SageMaker CreateTransformJob": { "Type": "Task", "Resource": "arn:aws:states:::sagemaker:createTransformJob.sync", "Arguments": { "ModelName": "SageMakerCreateTransformJobModel-9iFBKsYti9vr", "TransformInput": { "CompressionType": "None", "ContentType": "text/csv", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": "s3://amzn-s3-demo-source-bucket1/TransformJobDataInput.txt" } } }, "TransformOutput": { "S3OutputPath": "s3://amzn-s3-demo-source-bucket1/TransformJobOutputPath" }, "TransformResources": { "InstanceCount": 1, "InstanceType": "ml.m4.xlarge" }, "TransformJobName": "sfn-binary-classification-prediction" }, "Next": "ValidateOutput" }, ``` ## SageMaker AI Training Job Example The following includes a `Task` state that creates an Amazon SageMaker AI training job. ``` { "SageMaker CreateTrainingJob":{ "Type":"Task", "Resource":"arn:aws:states:::sagemaker:createTrainingJob.sync", "Arguments":{ "TrainingJobName":"search-model", "ResourceConfig":{ "InstanceCount":4, "InstanceType":"ml.c4.8xlarge", "VolumeSizeInGB":20 }, "HyperParameters":{ "mode":"batch_skipgram", "epochs":"5", "min_count":"5", "sampling_threshold":"0.0001", "learning_rate":"0.025", "window_size":"5", "vector_dim":"300", "negative_samples":"5", "batch_size":"11" }, "AlgorithmSpecification":{ "TrainingImage":"...", "TrainingInputMode":"File" }, "OutputDataConfig":{ "S3OutputPath":"s3://amzn-s3-demo-destination-bucket1/doc-search/model" }, "StoppingCondition":{ "MaxRuntimeInSeconds":100000 }, "RoleArn":"arn:aws:iam::account-id:role/docsearch-stepfunction-iam-role", "InputDataConfig":[ { "ChannelName":"train", "DataSource":{ "S3DataSource":{ "S3DataType":"S3Prefix", "S3Uri":"s3://amzn-s3-demo-destination-bucket1/doc-search/interim-data/training-data/", "S3DataDistributionType":"FullyReplicated" } } } ] }, "Retry":[ { "ErrorEquals":[ "SageMaker.AmazonSageMakerException" ], "IntervalSeconds":1, "MaxAttempts":100, "BackoffRate":1.1 }, { "ErrorEquals":[ "SageMaker.ResourceLimitExceededException" ], "IntervalSeconds":60, "MaxAttempts":5000, "BackoffRate":1 }, { "ErrorEquals":[ "States.Timeout" ], "IntervalSeconds":1, "MaxAttempts":5, "BackoffRate":1 } ], "Catch":[ { "ErrorEquals":[ "States.ALL" ], "Next":"Sagemaker Training Job Error" } ], "Next":"Delete Interim Data Job" } } ``` ## SageMaker AI Labeling Job Example The following includes a `Task` state that creates an Amazon SageMaker AI labeling job. ``` { "StartAt": "SageMaker CreateLabelingJob", "TimeoutSeconds": 3600, "States": { "SageMaker CreateLabelingJob": { "Type": "Task", "Resource": "arn:aws:states:::sagemaker:createLabelingJob.sync", "Arguments": { "HumanTaskConfig": { "AnnotationConsolidationConfig": { "AnnotationConsolidationLambdaArn": "arn:aws:lambda:region:123456789012:function:ACS-TextMultiClass" }, "NumberOfHumanWorkersPerDataObject": 1, "PreHumanTaskLambdaArn": "arn:aws:lambda:region:123456789012:function:PRE-TextMultiClass", "TaskDescription": "Classify the following text", "TaskKeywords": [ "tc", "Labeling" ], "TaskTimeLimitInSeconds": 300, "TaskTitle": "Classify short bits of text", "UiConfig": { "UiTemplateS3Uri": "s3://amzn-s3-demo-bucket/TextClassification.template" }, "WorkteamArn": "arn:aws:sagemaker:region:123456789012:workteam/private-crowd/ExampleTesting" }, "InputConfig": { "DataAttributes": { "ContentClassifiers": [ "FreeOfPersonallyIdentifiableInformation", "FreeOfAdultContent" ] }, "DataSource": { "S3DataSource": { "ManifestS3Uri": "s3://amzn-s3-demo-bucket/manifest.json" } } }, "LabelAttributeName": "Categories", "LabelCategoryConfigS3Uri": "s3://amzn-s3-demo-bucket/labelcategories.json", "LabelingJobName": "example-job-name", "OutputConfig": { "S3OutputPath": "s3://amzn-s3-demo-bucket/output" }, "RoleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole", "StoppingConditions": { "MaxHumanLabeledObjectCount": 10000, "MaxPercentageOfInputDatasetLabeled": 100 } }, "Next": "ValidateOutput" }, "ValidateOutput": { "Type": "Choice", "Choices": [ { "Next": "Success", "Condition": "{% $states.input.LabelingJobArn != '' %}" } ], "Default": "Fail" }, "Success": { "Type": "Succeed" }, "Fail": { "Type": "Fail", "Error": "InvalidOutput", "Cause": "Output is not what was expected. This could be due to a service outage or a misconfigured service integration." } } } ``` ## SageMaker AI Processing Job Example The following includes a `Task` state that creates an Amazon SageMaker AI processing job. ``` { "StartAt": "SageMaker CreateProcessingJob Sync", "TimeoutSeconds": 3600, "States": { "SageMaker CreateProcessingJob Sync": { "Type": "Task", "Resource": "arn:aws:states:::sagemaker:createProcessingJob.sync", "Arguments": { "AppSpecification": { "ImageUri": "737474898029.dkr.ecr.sa-east-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3" }, "ProcessingResources": { "ClusterConfig": { "InstanceCount": 1, "InstanceType": "ml.t3.medium", "VolumeSizeInGB": 10 } }, "RoleArn": "arn:aws:iam::account-id:role/SM-003-CreateProcessingJobAPIExecutionRole", "ProcessingJobName.$": "$.id" }, "Next": "ValidateOutput" }, "ValidateOutput": { "Type": "Choice", "Choices": [ { "Not": { "Variable": "$.ProcessingJobArn", "StringEquals": "" }, "Next": "Succeed" } ], "Default": "Fail" }, "Succeed": { "Type": "Succeed" }, "Fail": { "Type": "Fail", "Error": "InvalidConnectorOutput", "Cause": "Connector output is not what was expected. This could be due to a service outage or a misconfigured connector." } } } ``` ## IAM policies for calling Amazon SageMaker AI The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). **Note** For these examples, `roleArn` refers to the Amazon Resource Name (ARN) of the IAM role that SageMaker AI uses to access model artifacts and docker images for deployment on ML compute instances, or for batch transform jobs. For more information, see [Amazon SageMaker Roles](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html). ### `CreateTrainingJob` *Static resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTrainingJob", "sagemaker:DescribeTrainingJob", "sagemaker:StopTrainingJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:training-job/myJobName*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule" ] } ] } ``` ------ #### [ Request Response and Callback (.waitForTaskToken) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTrainingJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:training-job/myJobName*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } } ] } ``` ------ *Dynamic resources* ------ #### [ .sync or .waitForTaskToken ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTrainingJob", "sagemaker:DescribeTrainingJob", "sagemaker:StopTrainingJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:training-job/*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule" ] } ] } ``` ------ #### [ Request Response and Callback (.waitForTaskToken) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTrainingJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:training-job/*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } } ] } ``` ------ ### `CreateTransformJob` **Note** AWS Step Functions will not automatically create a policy for `CreateTransformJob` when you create a state machine that integrates with SageMaker AI. You must attach an inline policy to the created role based on one of the following IAM examples. *Static resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTransformJob", "sagemaker:DescribeTransformJob", "sagemaker:StopTransformJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:transform-job/myJobName*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule" ] } ] } ``` ------ #### [ Request Response and Callback (.waitForTaskToken) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTransformJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:transform-job/myJobName*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } } ] } ``` ------ *Dynamic resources* ------ #### [ Run a Job (.sync) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTransformJob", "sagemaker:DescribeTransformJob", "sagemaker:StopTransformJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:transform-job/*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule" ] } ] } ``` ------ #### [ Request Response and Callback (.waitForTaskToken) ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:CreateTransformJob" ], "Resource": [ "arn:aws:sagemaker:us-east-1:123456789012:transform-job/*" ] }, { "Effect": "Allow", "Action": [ "sagemaker:ListTags", "sagemaker:AddTags" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::123456789012:role/MyExampleRole" ], "Condition": { "StringEquals": { "iam:PassedToService": "sagemaker.amazonaws.com" } } } ] } ``` ------ # Publish messages to an Amazon SNS topic with Step Functions Learn how to use Step Functions to publish messages to an Amazon SNS topic. This page lists the supported Amazon SNS API actions and provides example `Task` states to publish message to Amazon SNS. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). **Key features of Optimized Amazon SNS integration** There are no specific optimizations for the [Request Response](connect-to-resource.md#connect-default) or [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token) integration patterns. The following includes a `Task` state that publishes to an Amazon Simple Notification Service (Amazon SNS) topic. ``` { "StartAt": "Publish to SNS", "States": { "Publish to SNS": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "Arguments": { "TopicArn": "arn:aws:sns:region:account-id:myTopic", "Message": "{% states.input.message %}", "MessageAttributes": { "my_attribute_no_1": { "DataType": "String", "StringValue": "value of my_attribute_no_1" }, "my_attribute_no_2": { "DataType": "String", "StringValue": "value of my_attribute_no_2" } } }, "End": true } } } ``` **Passing dynamic values**. You can modify the above example to dynamically pass an attribute from this JSON payload: ``` { "message": "Hello world", "SNSDetails": { "attribute1": "some value", "attribute2": "some other value", } } ``` The following sets values using JSONata expressions for the `StringValue` fields: ``` "MessageAttributes": { "my_attribute_no_1": { "DataType": "String", "StringValue": "{% $states.input.SNSDetails.attribute1 %}" }, "my_attribute_no_2": { "DataType": "String", "StringValue": "{% $states.input.SNSDetails.attribute2 %}" } ``` The following includes a `Task` state that publishes to an Amazon SNS topic, and then waits for the task token to be returned. See [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token). ``` { "StartAt":"Send message to SNS", "States":{ "Send message to SNS":{ "Type":"Task", "Resource":"arn:aws:states:::sns:publish.waitForTaskToken", "Arguments":{ "TopicArn":"arn:aws:sns:region:account-id:myTopic", "Message":{ "Input":"{% states.input.message %}", "TaskToken": "{% $states.context.Task.Token %}" } }, "End":true } } } ``` ## Optimized Amazon SNS APIs + [https://docs.aws.amazon.com/sns/latest/api/API_Publish.html](https://docs.aws.amazon.com/sns/latest/api/API_Publish.html) **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). ## IAM policies for calling Amazon SNS The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sns:Publish" ], "Resource": [ "arn:aws:sns:us-east-1:123456789012:myTopicName" ] } ] } ``` *Resources based on a Path, or publishing to `TargetArn` or `PhoneNumber`* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sns:Publish" ], "Resource": "*" } ] } ``` # Send messages to an Amazon SQS queue with Step Functions You can to send messages to an Amazon SQS queue using the following Amazon SQS API actions and example `Task` state code for Step Functions workflows. To learn about integrating with AWS services in Step Functions, see [Integrating services](integrate-services.md) and [Passing parameters to a service API in Step Functions](connect-parameters.md). To learn more about receiving messages in Amazon SQS, see [Receive and Delete Your Message](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/step-receive-delete-message.html) in the *Amazon Simple Queue Service Developer Guide*. The following sample includes a `Task` state (JSONata) that sends an Amazon Simple Queue Service (Amazon SQS) message with optional **MessageAttributes**: ``` { "StartAt": "Send to SQS", "States": { "Send to SQS": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage", "Arguments": { "QueueUrl": "https://sqs.us-east-1.amazonaws.com/account-id/myQueue", "MessageBody": "{% $states.input.message %}", "MessageAttributes": { "my_attribute_no_1": { "DataType": "String", "StringValue": "attribute1" }, "my_attribute_no_2": { "DataType": "String", "StringValue": "attribute2" } } }, "End": true } } } ``` The following state machine includes a `Task` state that publishes to an Amazon SQS queue, and then waits for the task token to be returned. See [Wait for a Callback with Task Token](connect-to-resource.md#connect-wait-token). ``` { "StartAt":"Send message to SQS", "States":{ "Send message to SQS":{ "Type":"Task", "Resource":"arn:aws:states:::sqs:sendMessage.waitForTaskToken", "Arguments":{ "QueueUrl":"https://sqs.us-east-1.amazonaws.com/account-id/myQueue", "MessageBody":{ "Input" : "{% $states.input.message %}", "MyTaskToken" : "{% $states.context.Task.Token %}" } }, "End":true } } } ``` ## Optimized Amazon SQS APIs + [https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html) **Parameters in Step Functions are expressed in PascalCase** Even if the native service API is in camelCase, for example the API action `startSyncExecution`, you specify parameters in PascalCase, such as: `StateMachineArn`. **Quota for input or result data** When sending or receiving data between services, the maximum input or result for a task is 256 KiB of data as a UTF-8 encoded string. See [Quotas related to state machine executions](service-quotas.md#service-limits-state-machine-executions). ## IAM policies for calling Amazon SQS The following example templates show how AWS Step Functions generates IAM policies based on the resources in your state machine definition. For more information, see [How Step Functions generates IAM policies for integrated services](service-integration-iam-templates.md) and [Discover service integration patterns in Step Functions](connect-to-resource.md). *Static resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:SendMessage" ], "Resource": [ "arn:aws:sqs:us-east-1:123456789012:myQueueName" ] } ] } ``` *Dynamic resources* **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:SendMessage" ], "Resource": "*" } ] } ``` # Start a new AWS Step Functions state machine from a running execution Step Functions integrates with its own API as a service integration. Learn how to use Step Functions to start a new execution of a state machine directly from the task state of a running execution. When building new workflows, use [nested workflow executions](concepts-nested-workflows.md) to reduce the complexity of your main workflows and to reuse common processes. **Key features of Optimized Step Functions integration** The [Run a Job (.sync)](connect-to-resource.md#connect-sync) integration pattern is available. For more information, see the following: + [Start from a Task](concepts-nested-workflows.md) + [Integrating services](integrate-services.md) + [Passing parameters to a service API in Step Functions](connect-parameters.md) ## Optimized Step Functions APIs + [https://docs.aws.amazon.com/step-functions/latest/apireference/API_StartExecution.html](https://docs.aws.amazon.com/step-functions/latest/apireference/API_StartExecution.html) ## Workflow Examples The following includes a `Task` state that starts an execution of another state machine and waits for it to complete. ``` { "Type":"Task", "Resource":"arn:aws:states:::states:startExecution.sync:2", "Arguments":{ "Input":{ "Comment": "Hello world!" }, "StateMachineArn":"arn:aws:states:region:account-id:stateMachine:HelloWorld", "Name":"ExecutionName" }, "End":true } ``` The following includes a `Task` state that starts an execution of another state machine. ``` { "Type":"Task", "Resource":"arn:aws:states:::states:startExecution", "Arguments":{ "Input":{ "Comment": "Hello world!" }, "StateMachineArn":"arn:aws:states:region:account-id:stateMachine:HelloWorld", "Name":"ExecutionName" }, "End":true } ``` The following includes a `Task` state that implements the [callback](connect-to-resource.md#connect-wait-token) service integration pattern. ``` { "Type":"Task", "Resource":"arn:aws:states:::states:startExecution.waitForTaskToken", "Arguments":{ "Input":{ "Comment": "Hello world!", "token": "{% $states.context.Task.Token %}" }, "StateMachineArn":"arn:aws:states:region:account-id:stateMachine:HelloWorld", "Name":"ExecutionName" }, "End":true } ``` To associate a nested workflow execution with the parent execution that started it, pass a specially named parameter that includes the execution ID pulled from the [Context object](input-output-contextobject.md). When starting a nested execution, use a parameter named `AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID`. Pass the execution ID and referencing the ID in the Context object with `$states.context.Execution.Id`. For more information, see [Accessing the Context object](input-output-contextobject.md#contextobject-access). ``` { "Type":"Task", "Resource":"arn:aws:states:::states:startExecution.sync", "Arguments":{ "Input":{ "Comment": "Hello world!", "AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID": "{% $states.context.Execution.Id %}" }, "StateMachineArn":"arn:aws:states:region:account-id:stateMachine:HelloWorld", "Name":"ExecutionName" }, "End":true } ``` Nested state machines return the following: | Resource | Output | | --- | --- | | startExecution.sync | String | | startExecution.sync:2 | JSON | Both will wait for the nested state machine to complete, but they return different `Output` formats. For example, if you create a Lambda function that returns the object `{ "MyKey": "MyValue" }`, you would get the following responses: For startExecution.sync: ``` { "Output": "{ \"MyKey\": \"MyValue\" }" } ``` For startExecution.sync:2: ``` { "Output": { "MyKey": "MyValue" } } ``` ### Configuring IAM permissions for nested state machines A parent state machine determines if a child state machine has completed execution using polling and events. Polling requires permission for `states:DescribeExecution` while events sent through EventBridge to Step Functions require permissions for `events:PutTargets`, `events:PutRule`, and `events:DescribeRule`. If these permissions are missing from your IAM role, there may be a delay before a parent state machine becomes aware of the completion of the child state machine's execution. For a state machine that calls `StartExecution` for a single nested workflow execution, use an IAM policy that limits permissions to that state machine. ## IAM policies for calling nested Step Functions workflows For a state machine that calls `StartExecution` for a single nested workflow execution, use an IAM policy that limits permissions to that state machine. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "states:StartExecution" ], "Resource": [ "arn:aws:states:us-east-1:123456789012:stateMachine:myStateMachineName" ] } ] } ``` For more information, see the following: + [Integrating services with Step Functions](integrate-services.md) + [Passing parameters to a service API in Step Functions](connect-parameters.md) + [Start a new AWS Step Functions state machine from a running execution](#connect-stepfunctions) ------ #### [ Synchronous ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "states:StartExecution" ], "Resource": [ "arn:aws:states:us-east-1:123456789012:stateMachine:stateMachineName" ] }, { "Effect": "Allow", "Action": [ "states:DescribeExecution", "states:StopExecution" ], "Resource": [ "arn:aws:states:us-east-1:123456789012:execution:myStateMachineName:*" ] }, { "Effect": "Allow", "Action": [ "events:PutTargets", "events:PutRule", "events:DescribeRule" ], "Resource": [ "arn:aws:events:us-east-1:123456789012:rule/StepFunctionsGetEventsForStepFunctionsExecutionRule" ] } ] } ``` ------ #### [ Asynchronous ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "states:StartExecution" ], "Resource": [ "arn:aws:states:us-east-1:123456789012:stateMachine:myStateMachineName" ] } ] } ``` ------ **ARN types required** In the policy for **Synchronous**, note that `states:StartExecution` requires a state machine ARN whereas `states:DescribeExecution` and `states:StopExecution` require an execution ARN. If you mistakenly combine all three actions, the JSON will be valid but the IAM policy will be incorrect. An incorrect policy can cause stuck workflows and/or access issues during workflow execution. For more information about nested workflow executions, see [Start workflow executions from a task state in Step Functions](concepts-nested-workflows.md).