通过 Amazon EventBridge 实现 EMR Serverless 的自动化
您可以使用 Amazon EventBridge 自动执行 AWS 服务 并自动响应系统事件,例如应用程序可用性问题或资源更改。Eventbridge 提供近乎实时的系统事件流,这些事件描述了 AWS 资源的变化。您可以编写简单规则来指示您关注的事件,并指示要在事件匹配规则时执行的自动化操作。通过 EventBridge,您可以自动:
-
调用 AWS Lambda 函数
-
将事件中继到 Amazon Kinesis Data Streams
-
激活 AWS Step Functions 状态机
-
通知 Amazon SNS 主题或 Amazon SQS 队列
例如,当您将 EventBridge 与 EMR Serverless 结合使用时,可在 ETL 作业成功时激活一个 AWS Lambda 函数,或在 ETL 作业失败时通知 Amazon SNS 主题。
EMR Serverless 会发出四种事件:
EMR Serverless EventBridge 事件示例
EMR Serverless 报告的事件为 source
分配值 aws.emr-serverless
,如以下示例所示。
应用程序状态更改事件
以下示例事件显示了处于 CREATING
状态的应用程序。
{ "version": "0", "id": "9fd3cf79-1ff1-b633-4dd9-34508dc1e660", "detail-type": "EMR Serverless Application State Change", "source": "aws.emr-serverless", "account": "123456789012", "time": "2022-05-31T21:16:31Z", "region": "us-east-1", "resources": [], "detail": { "applicationId": "00f1cbsc6anuij25", "applicationName": "3965ad00-8fba-4932-a6c8-ded32786fd42", "arn": "arn:aws:emr-serverless:us-east-1:111122223333:/applications/00f1cbsc6anuij25", "releaseLabel": "emr-6.6.0", "state": "CREATING", "type": "HIVE", "createdAt": "2022-05-31T21:16:31.547953Z", "updatedAt": "2022-05-31T21:16:31.547970Z", "autoStopConfig": { "enabled": true, "idleTimeout": 15 }, "autoStartConfig": { "enabled": true } } }
作业运行状态更改事件
以下示例事件显示了状态从 SCHEDULED
变为 RUNNING
的作业运行。
{ "version": "0", "id": "00df3ec6-5da1-36e6-ab71-20f0de68f8a0", "detail-type": "EMR Serverless Job Run State Change", "source": "aws.emr-serverless", "account": "123456789012", "time": "2022-05-31T21:07:42Z", "region": "us-east-1", "resources": [], "detail": { "jobRunId": "00f1cbn5g4bb0c01", "applicationId": "00f1982r1uukb925", "arn": "arn:aws:emr-serverless:us-east-1:123456789012:/applications/00f1982r1uukb925/jobruns/00f1cbn5g4bb0c01", "releaseLabel": "emr-6.6.0", "state": "RUNNING", "previousState": "SCHEDULED", "createdBy": "arn:aws:sts::123456789012:assumed-role/TestRole-402dcef3ad14993c15d28263f64381e4cda34775/6622b6233b6d42f59c25dd2637346242", "updatedAt": "2022-05-31T21:07:42.299487Z", "createdAt": "2022-05-31T21:07:25.325900Z" } }
作业运行重试事件
以下是作业运行重试事件的示例。
{ "version": "0", "id": "00df3ec6-5da1-36e6-ab71-20f0de68f8a0", "detail-type": "EMR Serverless Job Run Retry", "source": "aws.emr-serverless", "account": "123456789012", "time": "2022-05-31T21:07:42Z", "region": "us-east-1", "resources": [], "detail": { "jobRunId": "00f1cbn5g4bb0c01", "applicationId": "00f1982r1uukb925", "arn": "arn:aws:emr-serverless:us-east-1:123456789012:/applications/00f1982r1uukb925/jobruns/00f1cbn5g4bb0c01", "releaseLabel": "emr-6.6.0", "createdBy": "arn:aws:sts::123456789012:assumed-role/TestRole-402dcef3ad14993c15d28263f64381e4cda34775/6622b6233b6d42f59c25dd2637346242", "updatedAt": "2022-05-31T21:07:42.299487Z", "createdAt": "2022-05-31T21:07:25.325900Z", //Attempt Details "previousAttempt": 1, "previousAttemptState": "FAILED", "previousAttemptCreatedAt": "2022-05-31T21:07:25.325900Z", "previousAttemptEndedAt": "2022-05-31T21:07:30.325900Z", "newAttempt": 2, "newAttemptCreatedAt": "2022-05-31T21:07:30.325900Z" } }
作业资源利用率更新
以下示例事件显示了运行后转至最终状态的作业的最终资源利用率更新。
{ "version": "0", "id": "00df3ec6-5da1-36e6-ab71-20f0de68f8a0", "detail-type": "EMR Serverless Job Resource Utilization Update", "source": "aws.emr-serverless", "account": "123456789012", "time": "2022-05-31T21:07:42Z", "region": "us-east-1", "resources": [ "arn:aws:emr-serverless:us-east-1:123456789012:/applications/00f1982r1uukb925/jobruns/00f1cbn5g4bb0c01" ], "detail": { "applicationId": "00f1982r1uukb925", "jobRunId": "00f1cbn5g4bb0c01", "attempt": 1, "mode": "BATCH", "createdAt": "2022-05-31T21:07:25.325900Z", "startedAt": "2022-05-31T21:07:26.123Z", "calculatedFrom": "2022-05-31T21:07:42.299487Z", "calculatedTo": "2022-05-31T21:07:30.325900Z", "resourceUtilizationFinal": true, "resourceUtilizationForInterval": { "vCPUHour": 0.023, "memoryGBHour": 0.114, "storageGBHour": 0.228 }, "billedResourceUtilizationForInterval": { "vCPUHour": 0.067, "memoryGBHour": 0.333, "storageGBHour": 0 }, "totalResourceUtilization": { "vCPUHour": 0.023, "memoryGBHour": 0.114, "storageGBHour": 0.228 }, "totalBilledResourceUtilization": { "vCPUHour": 0.067, "memoryGBHour": 0.333, "storageGBHour": 0 } } }
仅当作业转至运行状态时,startAt 字段才会出现在事件中。