文件 AWS 開發套件範例 GitHub 儲存庫中有更多可用的 [AWS SDK 範例](https://github.com/awsdocs/aws-doc-sdk-examples)。
本文為英文版的機器翻譯版本，如內容有任何歧義或不一致之處，概以英文版為準。
# Amazon EMR AWS SDKs的程式碼範例
下列程式碼範例示範如何使用 Amazon EMR 搭配 AWS 軟體開發套件 (SDK)。
*Actions* 是大型程式的程式碼摘錄，必須在內容中執行。雖然動作會告訴您如何呼叫個別服務函數，但您可以在其相關情境中查看內容中的動作。
*案例*是向您展示如何呼叫服務中的多個函數或與其他 AWS 服務組合來完成特定任務的程式碼範例。
**其他資源**
+ **[Amazon EMR 管理指南](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html)** – Amazon EMR 的詳細資訊。
+ **[Amazon EMR API 參考](https://docs.aws.amazon.com/emr/latest/APIReference/Welcome.html)** – 所有可用 Amazon EMR 動作的詳細資訊。
+ **[AWS 開發人員中心](https://aws.amazon.com/developer/code-examples/?awsf.sdk-code-examples-product=product%23emr)** – 您可以依類別或全文搜尋篩選的程式碼範例。
+ **[AWS SDK 範例](https://github.com/awsdocs/aws-doc-sdk-examples)** – GitHub 儲存庫使用慣用語言的完整程式碼。包含設定和執行程式碼的指示。
**Contents**
+ [基本概念](emr_code_examples_basics.md)
+ [動作](emr_code_examples_actions.md)
+ [`AddJobFlowSteps`](emr_example_emr_AddJobFlowSteps_section.md)
+ [`DescribeCluster`](emr_example_emr_DescribeCluster_section.md)
+ [`DescribeStep`](emr_example_emr_DescribeStep_section.md)
+ [`ListSteps`](emr_example_emr_ListSteps_section.md)
+ [`RunJobFlow`](emr_example_emr_RunJobFlow_section.md)
+ [`TerminateJobFlows`](emr_example_emr_TerminateJobFlows_section.md)
+ [案例](emr_code_examples_scenarios.md)
+ [建立短期的 Amazon EMR 叢集，並執行一個步驟](emr_example_emr_Scenario_ShortLivedEmrCluster_section.md)
+ [執行 Shell 指令碼來安裝程式庫](emr_example_emr_Usage_InstallLibrariesWithSsm_section.md)
# Amazon EMR AWS SDKs的基本範例
下列程式碼範例示範如何搭配 AWS SDK 使用 Amazon EMR 的基本功能。
**Contents**
+ [動作](emr_code_examples_actions.md)
+ [`AddJobFlowSteps`](emr_example_emr_AddJobFlowSteps_section.md)
+ [`DescribeCluster`](emr_example_emr_DescribeCluster_section.md)
+ [`DescribeStep`](emr_example_emr_DescribeStep_section.md)
+ [`ListSteps`](emr_example_emr_ListSteps_section.md)
+ [`RunJobFlow`](emr_example_emr_RunJobFlow_section.md)
+ [`TerminateJobFlows`](emr_example_emr_TerminateJobFlows_section.md)
# Amazon EMR AWS SDKs的動作
下列程式碼範例示範如何使用 AWS SDKs 執行個別 Amazon EMR 動作。每個範例均包含 GitHub 的連結，您可以在連結中找到設定和執行程式碼的相關說明。
這些摘錄會呼叫 Amazon EMR API，是必須在內容中執行之大型程式的程式碼摘錄。您可以在 [使用 AWS SDKs Amazon EMR 案例](emr_code_examples_scenarios.md) 中查看內容中的動作。
下列範例僅包含最常使用的動作。如需完整清單，請參閱《[Amazon EMR API 參考](https://docs.aws.amazon.com/emr/latest/APIReference/Welcome.html)》。
**Topics**
+ [`AddJobFlowSteps`](emr_example_emr_AddJobFlowSteps_section.md)
+ [`DescribeCluster`](emr_example_emr_DescribeCluster_section.md)
+ [`DescribeStep`](emr_example_emr_DescribeStep_section.md)
+ [`ListSteps`](emr_example_emr_ListSteps_section.md)
+ [`RunJobFlow`](emr_example_emr_RunJobFlow_section.md)
+ [`TerminateJobFlows`](emr_example_emr_TerminateJobFlows_section.md)
# `AddJobFlowSteps` 搭配 AWS SDK 使用
下列程式碼範例示範如何使用 `AddJobFlowSteps`。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
新增 Spark 步驟，該步驟會在新增叢集後立即執行。
```
def add_step(cluster_id, name, script_uri, script_args, emr_client):
"""
Adds a job step to the specified cluster. This example adds a Spark
step, which is run by the cluster as soon as it is added.
:param cluster_id: The ID of the cluster.
:param name: The name of the step.
:param script_uri: The URI where the Python script is stored.
:param script_args: Arguments to pass to the Python script.
:param emr_client: The Boto3 EMR client object.
:return: The ID of the newly added step.
"""
try:
response = emr_client.add_job_flow_steps(
JobFlowId=cluster_id,
Steps=[
{
"Name": name,
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
script_uri,
*script_args,
],
},
}
],
)
step_id = response["StepIds"][0]
logger.info("Started step with ID %s", step_id)
except ClientError:
logger.exception("Couldn't start step %s with URI %s.", name, script_uri)
raise
else:
return step_id
```
在叢集上執行 Amazon EMR 檔案系統 (EMRFS) 命令，以做為任務步驟。這可用於自動化叢集上的 EMRFS 命令，而不是透過 SSH 連線手動執行命令。
```
import boto3
from botocore.exceptions import ClientError
def add_emrfs_step(command, bucket_url, cluster_id, emr_client):
"""
Add an EMRFS command as a job flow step to an existing cluster.
:param command: The EMRFS command to run.
:param bucket_url: The URL of a bucket that contains tracking metadata.
:param cluster_id: The ID of the cluster to update.
:param emr_client: The Boto3 Amazon EMR client object.
:return: The ID of the added job flow step. Status can be tracked by calling
the emr_client.describe_step() function.
"""
job_flow_step = {
"Name": "Example EMRFS Command Step",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": ["/usr/bin/emrfs", command, bucket_url],
},
}
try:
response = emr_client.add_job_flow_steps(
JobFlowId=cluster_id, Steps=[job_flow_step]
)
step_id = response["StepIds"][0]
print(f"Added step {step_id} to cluster {cluster_id}.")
except ClientError:
print(f"Couldn't add a step to cluster {cluster_id}.")
raise
else:
return step_id
def usage_demo():
emr_client = boto3.client("emr")
# Assumes the first waiting cluster has EMRFS enabled and has created metadata
# with the default name of 'EmrFSMetadata'.
cluster = emr_client.list_clusters(ClusterStates=["WAITING"])["Clusters"][0]
add_emrfs_step(
"sync", "s3://elasticmapreduce/samples/cloudfront", cluster["Id"], emr_client
)
if __name__ == "__main__":
usage_demo()
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [AddJobFlowSteps](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/AddJobFlowSteps)。
------
#### [ SAP ABAP ]
**適用於 SAP ABAP 的開發套件**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples)中設定和執行。
```
TRY.
" Build args list for Spark submit
DATA lt_args TYPE /aws1/cl_emrxmlstringlist_w=>tt_xmlstringlist.
APPEND NEW /aws1/cl_emrxmlstringlist_w( 'spark-submit' ) TO lt_args.
APPEND NEW /aws1/cl_emrxmlstringlist_w( '--deploy-mode' ) TO lt_args.
APPEND NEW /aws1/cl_emrxmlstringlist_w( 'cluster' ) TO lt_args.
APPEND NEW /aws1/cl_emrxmlstringlist_w( iv_script_uri ) TO lt_args.
APPEND LINES OF it_script_args TO lt_args.
" Create step configuration
DATA(lo_hadoop_jar_step) = NEW /aws1/cl_emrhadoopjarstepcfg(
iv_jar = 'command-runner.jar'
it_args = lt_args
).
DATA(lo_step_config) = NEW /aws1/cl_emrstepconfig(
iv_name = iv_name
iv_actiononfailure = 'CONTINUE'
io_hadoopjarstep = lo_hadoop_jar_step
).
DATA lt_steps TYPE /aws1/cl_emrstepconfig=>tt_stepconfiglist.
APPEND lo_step_config TO lt_steps.
DATA(lo_result) = lo_emr->addjobflowsteps(
iv_jobflowid = iv_cluster_id
it_steps = lt_steps
).
" Get first step ID
DATA(lt_step_ids) = lo_result->get_stepids( ).
READ TABLE lt_step_ids INDEX 1 INTO DATA(lo_step_id_obj).
IF sy-subrc = 0.
ov_step_id = lo_step_id_obj->get_value( ).
MESSAGE |Step added with ID { ov_step_id }| TYPE 'I'.
ENDIF.
CATCH /aws1/cx_emrinternalservererr INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ 如需 API 詳細資訊，請參閱《適用於 *AWS SAP ABAP 的 SDK API 參考*》中的 [AddJobFlowSteps](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html)。
------
# `DescribeCluster` 搭配 AWS SDK 或 CLI 使用
下列程式碼範例示範如何使用 `DescribeCluster`。
------
#### [ CLI ]
**AWS CLI**
命令：
```
aws emr describe-cluster --cluster-id j-XXXXXXXX
```
輸出：
```
For release-label based uniform instance groups cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1436475075.199,
"CreationDateTime": 1436474656.563,
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"ServiceAccessSecurityGroup": "sg-xxxxxxxx",
"EmrManagedMasterSecurityGroup": "sg-xxxxxxxx",
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2KeyName": "myKey",
"Ec2AvailabilityZone": "us-east-1c",
"EmrManagedSlaveSecurityGroup": "sg-yyyyyyyyy"
},
"Name": "My Cluster",
"ServiceRole": "EMR_DefaultRole",
"Tags": [],
"TerminationProtected": true,
"UnhealthyNodeReplacement": true,
"ReleaseLabel": "emr-4.0.0",
"NormalizedInstanceHours": 96,
"InstanceGroups": [
{
"RequestedInstanceCount": 2,
"Status": {
"Timeline": {
"ReadyDateTime": 1436475074.245,
"CreationDateTime": 1436474656.564,
"EndDateTime": 1436638158.387
},
"State": "RUNNING",
"StateChangeReason": {
"Message": "",
}
},
"Name": "CORE",
"InstanceGroupType": "CORE",
"Id": "ig-YYYYYYY",
"Configurations": [],
"InstanceType": "m3.large",
"Market": "ON_DEMAND",
"RunningInstanceCount": 2
},
{
"RequestedInstanceCount": 1,
"Status": {
"Timeline": {
"ReadyDateTime": 1436475074.245,
"CreationDateTime": 1436474656.564,
"EndDateTime": 1436638158.387
},
"State": "RUNNING",
"StateChangeReason": {
"Message": "",
}
},
"Name": "MASTER",
"InstanceGroupType": "MASTER",
"Id": "ig-XXXXXXXXX",
"Configurations": [],
"InstanceType": "m3.large",
"Market": "ON_DEMAND",
"RunningInstanceCount": 1
}
],
"Applications": [
{
"Name": "Hadoop"
}
],
"VisibleToAllUsers": true,
"BootstrapActions": [],
"MasterPublicDnsName": "ec2-54-147-144-78.compute-1.amazonaws.com",
"AutoTerminate": false,
"Id": "j-XXXXXXXX",
"Configurations": [
{
"Properties": {
"fs.s3.consistent.retryPeriodSeconds": "20",
"fs.s3.enableServerSideEncryption": "true",
"fs.s3.consistent": "false",
"fs.s3.consistent.retryCount": "2"
},
"Classification": "emrfs-site"
}
]
}
}
For release-label based instance fleet cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1487897289.705,
"CreationDateTime": 1487896933.942
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"EmrManagedMasterSecurityGroup": "sg-xxxxx",
"RequestedEc2AvailabilityZones": [],
"RequestedEc2SubnetIds": [],
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2AvailabilityZone": "us-east-1a",
"EmrManagedSlaveSecurityGroup": "sg-xxxxx"
},
"Name": "My Cluster",
"ServiceRole": "EMR_DefaultRole",
"Tags": [],
"TerminationProtected": false,
"UnhealthyNodeReplacement": false,
"ReleaseLabel": "emr-5.2.0",
"NormalizedInstanceHours": 472,
"InstanceCollectionType": "INSTANCE_FLEET",
"InstanceFleets": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1487897212.74,
"CreationDateTime": 1487896933.948
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"ProvisionedSpotCapacity": 1,
"Name": "MASTER",
"InstanceFleetType": "MASTER",
"LaunchSpecifications": {
"SpotSpecification": {
"TimeoutDurationMinutes": 60,
"TimeoutAction": "TERMINATE_CLUSTER"
}
},
"TargetSpotCapacity": 1,
"ProvisionedOnDemandCapacity": 0,
"InstanceTypeSpecifications": [
{
"BidPrice": "0.5",
"InstanceType": "m3.xlarge",
"WeightedCapacity": 1
}
],
"Id": "if-xxxxxxx",
"TargetOnDemandCapacity": 0
}
],
"Applications": [
{
"Version": "2.7.3",
"Name": "Hadoop"
}
],
"ScaleDownBehavior": "TERMINATE_AT_INSTANCE_HOUR",
"VisibleToAllUsers": true,
"BootstrapActions": [],
"MasterPublicDnsName": "ec2-xxx-xx-xxx-xx.compute-1.amazonaws.com",
"AutoTerminate": false,
"Id": "j-xxxxx",
"Configurations": []
}
}
For ami based uniform instance group cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1399400564.432,
"CreationDateTime": 1399400268.62
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2AvailabilityZone": "us-east-1c"
},
"Name": "My Cluster",
"Tags": [],
"TerminationProtected": true,
"UnhealthyNodeReplacement": true,
"RunningAmiVersion": "2.5.4",
"InstanceGroups": [
{
"RequestedInstanceCount": 1,
"Status": {
"Timeline": {
"ReadyDateTime": 1399400558.848,
"CreationDateTime": 1399400268.621
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"Name": "Master instance group",
"InstanceGroupType": "MASTER",
"InstanceType": "m1.small",
"Id": "ig-ABCD",
"Market": "ON_DEMAND",
"RunningInstanceCount": 1
},
{
"RequestedInstanceCount": 2,
"Status": {
"Timeline": {
"ReadyDateTime": 1399400564.439,
"CreationDateTime": 1399400268.621
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"Name": "Core instance group",
"InstanceGroupType": "CORE",
"InstanceType": "m1.small",
"Id": "ig-DEF",
"Market": "ON_DEMAND",
"RunningInstanceCount": 2
}
],
"Applications": [
{
"Version": "1.0.3",
"Name": "hadoop"
}
],
"BootstrapActions": [],
"VisibleToAllUsers": false,
"RequestedAmiVersion": "2.4.2",
"LogUri": "s3://myLogUri/",
"AutoTerminate": false,
"Id": "j-XXXXXXXX"
}
}
```
+ 如需 API 詳細資訊，請參閱《AWS CLI 命令參考》**中的 [DescribeCluster](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/describe-cluster.html)。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
```
def describe_cluster(cluster_id, emr_client):
"""
Gets detailed information about a cluster.
:param cluster_id: The ID of the cluster to describe.
:param emr_client: The Boto3 EMR client object.
:return: The retrieved cluster information.
"""
try:
response = emr_client.describe_cluster(ClusterId=cluster_id)
cluster = response["Cluster"]
logger.info("Got data for cluster %s.", cluster["Name"])
except ClientError:
logger.exception("Couldn't get data for cluster %s.", cluster_id)
raise
else:
return cluster
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [DescribeCluster](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/DescribeCluster)。
------
#### [ SAP ABAP ]
**適用於 SAP ABAP 的開發套件**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples)中設定和執行。
```
TRY.
oo_result = lo_emr->describecluster(
iv_clusterid = iv_cluster_id
).
DATA(lo_cluster) = oo_result->get_cluster( ).
DATA(lv_cluster_name) = lo_cluster->get_name( ).
MESSAGE |Retrieved cluster information for { lv_cluster_name }| TYPE 'I'.
CATCH /aws1/cx_emrinternalserverex INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrinvalidrequestex INTO DATA(lo_invalid_error).
lv_error = lo_invalid_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ 如需 API 詳細資訊，請參閱《適用於 *AWS SAP ABAP 的 SDK API 參考*》中的 [DescribeCluster](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html)。
------
# `DescribeStep` 搭配 AWS SDK 或 CLI 使用
下列程式碼範例示範如何使用 `DescribeStep`。
------
#### [ CLI ]
**AWS CLI**
下列命令描述叢集中步驟 ID 為 `s-3LZC0QUT43AM`，且叢集 ID 為 `j-3SD91U2E1L2QX` 的步驟：
```
aws emr describe-step --cluster-id j-3SD91U2E1L2QX --step-id s-3LZC0QUT43AM
```
輸出：
```
{
"Step": {
"Status": {
"Timeline": {
"EndDateTime": 1433200470.481,
"CreationDateTime": 1433199926.597,
"StartDateTime": 1433200404.959
},
"State": "COMPLETED",
"StateChangeReason": {}
},
"Config": {
"Args": [
"s3://us-west-2.elasticmapreduce/libs/hive/hive-script",
"--base-path",
"s3://us-west-2.elasticmapreduce/libs/hive/",
"--install-hive",
"--hive-versions",
"0.13.1"
],
"Jar": "s3://us-west-2.elasticmapreduce/libs/script-runner/script-runner.jar",
"Properties": {}
},
"Id": "s-3LZC0QUT43AM",
"ActionOnFailure": "TERMINATE_CLUSTER",
"Name": "Setup hive"
}
}
```
+ 如需 API 詳細資訊，請參閱《AWS CLI 命令參考》**中的 [DescribeStep](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/describe-step.html)。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
```
def describe_step(cluster_id, step_id, emr_client):
"""
Gets detailed information about the specified step, including the current state of
the step.
:param cluster_id: The ID of the cluster.
:param step_id: The ID of the step.
:param emr_client: The Boto3 EMR client object.
:return: The retrieved information about the specified step.
"""
try:
response = emr_client.describe_step(ClusterId=cluster_id, StepId=step_id)
step = response["Step"]
logger.info("Got data for step %s.", step_id)
except ClientError:
logger.exception("Couldn't get data for step %s.", step_id)
raise
else:
return step
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [DescribeStep](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/DescribeStep)。
------
#### [ SAP ABAP ]
**適用於 SAP ABAP 的開發套件**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples)中設定和執行。
```
TRY.
oo_result = lo_emr->describestep(
iv_clusterid = iv_cluster_id
iv_stepid = iv_step_id
).
DATA(lo_step) = oo_result->get_step( ).
DATA(lv_step_name) = lo_step->get_name( ).
MESSAGE |Retrieved step information for { lv_step_name }| TYPE 'I'.
CATCH /aws1/cx_emrinternalserverex INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrinvalidrequestex INTO DATA(lo_invalid_error).
lv_error = lo_invalid_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ 如需 API 詳細資訊，請參閱《適用於 *AWS SAP ABAP 的 SDK API 參考*》中的 [DescribeStep](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html)。
------
# `ListSteps` 搭配 AWS SDK 或 CLI 使用
下列程式碼範例示範如何使用 `ListSteps`。
------
#### [ CLI ]
**AWS CLI**
下列命令會列出叢集中叢集 ID 為 `j-3SD91U2E1L2QX` 的所有步驟：
```
aws emr list-steps --cluster-id j-3SD91U2E1L2QX
```
+ 如需 API 詳細資訊，請參閱《AWS CLI 命令參考》**中的 [ListSteps](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-steps.html)。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
```
def list_steps(cluster_id, emr_client):
"""
Gets a list of steps for the specified cluster. In this example, all steps are
returned, including completed and failed steps.
:param cluster_id: The ID of the cluster.
:param emr_client: The Boto3 EMR client object.
:return: The list of steps for the specified cluster.
"""
try:
response = emr_client.list_steps(ClusterId=cluster_id)
steps = response["Steps"]
logger.info("Got %s steps for cluster %s.", len(steps), cluster_id)
except ClientError:
logger.exception("Couldn't get steps for cluster %s.", cluster_id)
raise
else:
return steps
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [ListSteps](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/ListSteps)。
------
#### [ SAP ABAP ]
**適用於 SAP ABAP 的開發套件**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples)中設定和執行。
```
TRY.
oo_result = lo_emr->liststeps(
iv_clusterid = iv_cluster_id
).
DATA(lt_steps) = oo_result->get_steps( ).
DATA(lv_step_count) = lines( lt_steps ).
MESSAGE |Retrieved { lv_step_count } steps for cluster| TYPE 'I'.
CATCH /aws1/cx_emrinternalserverex INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrinvalidrequestex INTO DATA(lo_invalid_error).
lv_error = lo_invalid_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ 如需 API 詳細資訊，請參閱《適用於 *AWS SAP ABAP 的 SDK API 參考*》中的 [ListSteps](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html)。
------
# `RunJobFlow` 搭配 AWS SDK 使用
下列程式碼範例示範如何使用 `RunJobFlow`。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
```
def run_job_flow(
name,
log_uri,
keep_alive,
applications,
job_flow_role,
service_role,
security_groups,
steps,
emr_client,
):
"""
Runs a job flow with the specified steps. A job flow creates a cluster of
instances and adds steps to be run on the cluster. Steps added to the cluster
are run as soon as the cluster is ready.
This example uses the 'emr-5.30.1' release. A list of recent releases can be
found here:
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html.
:param name: The name of the cluster.
:param log_uri: The URI where logs are stored. This can be an Amazon S3 bucket URL,
such as 's3://my-log-bucket'.
:param keep_alive: When True, the cluster is put into a Waiting state after all
steps are run. When False, the cluster terminates itself when
the step queue is empty.
:param applications: The applications to install on each instance in the cluster,
such as Hive or Spark.
:param job_flow_role: The IAM role assumed by the cluster.
:param service_role: The IAM role assumed by the service.
:param security_groups: The security groups to assign to the cluster instances.
Amazon EMR adds all needed rules to these groups, so
they can be empty if you require only the default rules.
:param steps: The job flow steps to add to the cluster. These are run in order
when the cluster is ready.
:param emr_client: The Boto3 EMR client object.
:return: The ID of the newly created cluster.
"""
try:
response = emr_client.run_job_flow(
Name=name,
LogUri=log_uri,
ReleaseLabel="emr-5.30.1",
Instances={
"MasterInstanceType": "m5.xlarge",
"SlaveInstanceType": "m5.xlarge",
"InstanceCount": 3,
"KeepJobFlowAliveWhenNoSteps": keep_alive,
"EmrManagedMasterSecurityGroup": security_groups["manager"].id,
"EmrManagedSlaveSecurityGroup": security_groups["worker"].id,
},
Steps=[
{
"Name": step["name"],
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
step["script_uri"],
*step["script_args"],
],
},
}
for step in steps
],
Applications=[{"Name": app} for app in applications],
JobFlowRole=job_flow_role.name,
ServiceRole=service_role.name,
EbsRootVolumeSize=10,
VisibleToAllUsers=True,
)
cluster_id = response["JobFlowId"]
logger.info("Created cluster %s.", cluster_id)
except ClientError:
logger.exception("Couldn't create cluster.")
raise
else:
return cluster_id
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [RunJobFlow](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/RunJobFlow)。
------
#### [ SAP ABAP ]
**適用於 SAP ABAP 的開發套件**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples)中設定和執行。
```
TRY.
" Create instances configuration
DATA(lo_instances) = NEW /aws1/cl_emrjobflowinstsconfig(
iv_masterinstancetype = 'm5.xlarge'
iv_slaveinstancetype = 'm5.xlarge'
iv_instancecount = 3
iv_keepjobflowalivewhennos00 = iv_keep_alive
iv_emrmanagedmastersecgroup = iv_primary_sec_grp
iv_emrmanagedslavesecgroup = iv_secondary_sec_grp
).
DATA(lo_result) = lo_emr->runjobflow(
iv_name = iv_name
iv_loguri = iv_log_uri
iv_releaselabel = 'emr-5.30.1'
io_instances = lo_instances
it_steps = it_steps
it_applications = it_applications
iv_jobflowrole = iv_job_flow_role
iv_servicerole = iv_service_role
iv_ebsrootvolumesize = 10
iv_visibletoallusers = abap_true
).
ov_cluster_id = lo_result->get_jobflowid( ).
MESSAGE 'EMR cluster created successfully.' TYPE 'I'.
CATCH /aws1/cx_emrinternalservererr INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrclientexc INTO DATA(lo_client_error).
lv_error = lo_client_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ 如需 API 詳細資訊，請參閱《適用於 *AWS SAP ABAP 的 SDK API 參考*》中的 [RunJobFlow](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html)。
------
# `TerminateJobFlows` 搭配 AWS SDK 使用
下列程式碼範例示範如何使用 `TerminateJobFlows`。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
```
def terminate_cluster(cluster_id, emr_client):
"""
Terminates a cluster. This terminates all instances in the cluster and cannot
be undone. Any data not saved elsewhere, such as in an Amazon S3 bucket, is lost.
:param cluster_id: The ID of the cluster to terminate.
:param emr_client: The Boto3 EMR client object.
"""
try:
emr_client.terminate_job_flows(JobFlowIds=[cluster_id])
logger.info("Terminated cluster %s.", cluster_id)
except ClientError:
logger.exception("Couldn't terminate cluster %s.", cluster_id)
raise
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [TerminateJobFlows](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/TerminateJobFlows)。
------
#### [ SAP ABAP ]
**適用於 SAP ABAP 的開發套件**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples)中設定和執行。
```
TRY.
DATA lt_cluster_ids TYPE /aws1/cl_emrxmlstringlist_w=>tt_xmlstringlist.
APPEND NEW /aws1/cl_emrxmlstringlist_w( iv_cluster_id ) TO lt_cluster_ids.
lo_emr->terminatejobflows(
it_jobflowids = lt_cluster_ids
).
MESSAGE 'EMR cluster terminated successfully.' TYPE 'I'.
CATCH /aws1/cx_emrinternalservererr INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ 如需 API 詳細資訊，請參閱《適用於 *AWS SAP ABAP 的 SDK API 參考*》中的 [TerminateJobFlows](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html)。
------
# 使用 AWS SDKs Amazon EMR 案例
下列程式碼範例示範如何在 Amazon EMR AWS SDKs 中實作常見案例。這些案例示範如何呼叫 Amazon EMR 中的多個函數，或與其他 AWS 服務結合，藉以完成特定任務。每個案例均包含完整原始碼的連結，您可在連結中找到如何設定和執行程式碼的相關指示。
案例的目標是獲得中等水平的經驗，協助您了解內容中的服務動作。
**Topics**
+ [建立短期的 Amazon EMR 叢集，並執行一個步驟](emr_example_emr_Scenario_ShortLivedEmrCluster_section.md)
+ [執行 Shell 指令碼來安裝程式庫](emr_example_emr_Usage_InstallLibrariesWithSsm_section.md)
# 建立短期 Amazon EMR 叢集並使用 AWS SDK 執行步驟
下列程式碼範例顯示如何建立短期 Amazon EMR 叢集，該叢集會執行一個步驟並在步驟完成後自動終止。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
建立短期的 Amazon EMR 叢集，該叢集使用 Apache Spark 預估 pi 的值，以平行化大量計算。該任務將輸出檔案寫入 Amazon EMR 日誌和 Amazon Simple Storage Service (Amazon S3) 儲存貯體。叢集會在完成任務後自行終止。
+ 建立 Amazon S3 儲存貯體並上傳任務指令碼。
+ Create AWS Identity and Access Management (IAM) 角色。
+ 建立 Amazon Elastic Compute Cloud (Amazon EC2) 安全群組。
+ 建立短期叢集並執行單個任務步驟。
這個範例在 GitHub 上的檢視效果最佳。如需完整的原始碼和如何設定及執行的指示，請參閱 [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr) 上的完整範例。
**此範例中使用的服務**
+ Amazon EMR
------
# 執行 shell 指令碼，使用 AWS SDK 在 Amazon EMR 執行個體上安裝程式庫
下列程式碼範例示範如何使用 AWS Systems Manager 在安裝其他程式庫的 Amazon EMR 執行個體上執行 shell 指令碼。如此一來，您就可以自動化執行個體管理，而不是透過 SSH 連線手動執行命令。
------
#### [ Python ]
**適用於 Python 的 SDK (Boto3)**
GitHub 上提供更多範例。尋找完整範例，並了解如何在 [AWS 程式碼範例儲存庫](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples)中設定和執行。
```
import argparse
import time
import boto3
def install_libraries_on_core_nodes(cluster_id, script_path, emr_client, ssm_client):
"""
Copies and runs a shell script on the core nodes in the cluster.
:param cluster_id: The ID of the cluster.
:param script_path: The path to the script, typically an Amazon S3 object URL.
:param emr_client: The Boto3 Amazon EMR client.
:param ssm_client: The Boto3 AWS Systems Manager client.
"""
core_nodes = emr_client.list_instances(
ClusterId=cluster_id, InstanceGroupTypes=["CORE"]
)["Instances"]
core_instance_ids = [node["Ec2InstanceId"] for node in core_nodes]
print(f"Found core instances: {core_instance_ids}.")
commands = [
# Copy the shell script from Amazon S3 to each node instance.
f"aws s3 cp {script_path} /home/hadoop",
# Run the shell script to install libraries on each node instance.
"bash /home/hadoop/install_libraries.sh",
]
for command in commands:
print(f"Sending '{command}' to core instances...")
command_id = ssm_client.send_command(
InstanceIds=core_instance_ids,
DocumentName="AWS-RunShellScript",
Parameters={"commands": [command]},
TimeoutSeconds=3600,
)["Command"]["CommandId"]
while True:
# Verify the previous step succeeded before running the next step.
cmd_result = ssm_client.list_commands(CommandId=command_id)["Commands"][0]
if cmd_result["StatusDetails"] == "Success":
print(f"Command succeeded.")
break
elif cmd_result["StatusDetails"] in ["Pending", "InProgress"]:
print(f"Command status is {cmd_result['StatusDetails']}, waiting...")
time.sleep(10)
else:
print(f"Command status is {cmd_result['StatusDetails']}, quitting.")
raise RuntimeError(
f"Command {command} failed to run. "
f"Details: {cmd_result['StatusDetails']}"
)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("cluster_id", help="The ID of the cluster.")
parser.add_argument("script_path", help="The path to the script in Amazon S3.")
args = parser.parse_args()
emr_client = boto3.client("emr")
ssm_client = boto3.client("ssm")
install_libraries_on_core_nodes(
args.cluster_id, args.script_path, emr_client, ssm_client
)
if __name__ == "__main__":
main()
```
+ 如需 API 詳細資訊，請參閱《AWS SDK for Python (Boto3) API 參考》**中的 [ListInstances](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/ListInstances)。
------