Class: Aws::EMR::Client
- Inherits:
-
Seahorse::Client::Base
- Object
- Seahorse::Client::Base
- Aws::EMR::Client
- Includes:
- ClientStubs
- Defined in:
- gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb
Overview
An API client for EMR. To construct a client, you need to configure a :region and :credentials.
client = Aws::EMR::Client.new(
region: region_name,
credentials: credentials,
# ...
)
For details on configuring region and credentials see the developer guide.
See #initialize for a full list of supported configuration options.
Instance Attribute Summary
Attributes inherited from Seahorse::Client::Base
API Operations collapse
-
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
-
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
-
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster.
-
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource, such as a cluster or an Amazon EMR Studio.
-
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster.
-
#create_persistent_app_ui(params = {}) ⇒ Types::CreatePersistentAppUIOutput
Creates a persistent application user interface.
-
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
-
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
-
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by
StudioId, and applies a session policy to refine Studio permissions for that user or group. -
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
-
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
-
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
-
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
-
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed.
-
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
-
#describe_persistent_app_ui(params = {}) ⇒ Types::DescribePersistentAppUIOutput
Describes a persistent application user interface.
-
#describe_release_label(params = {}) ⇒ Types::DescribeReleaseLabelOutput
Provides Amazon EMR release label details, such as the releases available the Region where the API request is run, and the available applications for a specific Amazon EMR release label.
-
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
-
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
-
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
-
#get_auto_termination_policy(params = {}) ⇒ Types::GetAutoTerminationPolicyOutput
Returns the auto-termination policy for an Amazon EMR cluster.
-
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your Amazon Web Services account in the current Region.
-
#get_cluster_session_credentials(params = {}) ⇒ Types::GetClusterSessionCredentialsOutput
Provides temporary, HTTP basic credentials that are associated with a given runtime IAM role and used by a cluster with fine-grained access control activated.
-
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
-
#get_on_cluster_app_ui_presigned_url(params = {}) ⇒ Types::GetOnClusterAppUIPresignedURLOutput
The presigned URL properties for the cluster's application user interface.
-
#get_persistent_app_ui_presigned_url(params = {}) ⇒ Types::GetPersistentAppUIPresignedURLOutput
The presigned URL properties for the cluster's application user interface.
-
#get_session(params = {}) ⇒ Types::GetSessionOutput
Returns detailed information about a session.
-
#get_session_endpoint(params = {}) ⇒ Types::GetSessionEndpointOutput
Returns the Spark Connect endpoint URL and a time-limited authentication token for the specified session.
-
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
-
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
-
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this Amazon Web Services account.
-
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
-
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
-
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active Amazon EC2 instances and Amazon EC2 instances terminated in the last 30 days, up to a maximum of 2,000.
-
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions.
-
#list_release_labels(params = {}) ⇒ Types::ListReleaseLabelsOutput
Retrieves release labels of Amazon EMR services in the Region where the API is called.
-
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names.
-
#list_sessions(params = {}) ⇒ Types::ListSessionsOutput
Lists the sessions on a cluster.
-
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you specify
stepIdswith the request or filter byStepStates. -
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the Amazon EMR Studio specified by
StudioId. -
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the Amazon Web Services account.
-
#list_supported_instance_types(params = {}) ⇒ Types::ListSupportedInstanceTypesOutput
A list of the instance types that Amazon EMR supports.
-
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
-
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID.
-
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group.
-
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster.
-
#put_auto_termination_policy(params = {}) ⇒ Struct
Auto-termination is supported in Amazon EMR releases 5.30.0 and 6.1.0 and later. -
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your Amazon Web Services account in the current Region.
-
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster.
-
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an Amazon EMR cluster.
-
#remove_auto_termination_policy(params = {}) ⇒ Struct
Removes an auto-termination policy from an Amazon EMR cluster.
-
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified Amazon EMR cluster.
-
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource, such as a cluster or Amazon EMR Studio.
-
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow).
-
#set_keep_job_flow_alive_when_no_steps(params = {}) ⇒ Struct
You can use the
SetKeepJobFlowAliveWhenNoStepsto configure a cluster (job flow) to terminate after the step execution, i.e., all your steps are executed. -
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the Amazon EC2 instances in the cluster cannot be terminated by user intervention, an API call, or in the event of a job-flow error.
-
#set_unhealthy_node_replacement(params = {}) ⇒ Struct
Specify whether to enable unhealthy node replacement, which lets Amazon EMR gracefully replace core nodes on a cluster if any nodes become unhealthy.
-
#set_visible_to_all_users(params = {}) ⇒ Struct
The SetVisibleToAllUsers parameter is no longer supported.
-
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
-
#start_session(params = {}) ⇒ Types::StartSessionOutput
Creates and starts a new Spark Connect session on the specified cluster.
-
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
-
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down.
-
#terminate_session(params = {}) ⇒ Types::TerminateSessionOutput
Terminates an active session.
-
#update_studio(params = {}) ⇒ Struct
Updates an Amazon EMR Studio configuration, including attributes such as name, description, and subnets.
-
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
Instance Method Summary collapse
-
#initialize(options) ⇒ Client
constructor
A new instance of Client.
-
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
Methods included from ClientStubs
#api_requests, #stub_data, #stub_responses
Methods inherited from Seahorse::Client::Base
add_plugin, api, clear_plugins, define, new, #operation_names, plugins, remove_plugin, set_api, set_plugins
Methods included from Seahorse::Client::HandlerBuilder
#handle, #handle_request, #handle_response
Constructor Details
#initialize(options) ⇒ Client
Returns a new instance of Client.
486 487 488 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 486 def initialize(*args) super end |
Instance Method Details
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
600 601 602 603 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 600 def add_instance_fleet(params = {}, = {}) req = build_request(:add_instance_fleet, params) req.send_request() end |
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
710 711 712 713 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 710 def add_instance_groups(params = {}, = {}) req = build_request(:add_instance_groups, params) req.send_request() end |
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster. A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using SSH to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop.
A step specifies the location of a JAR file stored either on the master node of the cluster or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step.
Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully.
You can only add steps to a cluster that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.
HadoopJarStep object cannot exceed a
total of 10240 characters.
803 804 805 806 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 803 def add_job_flow_steps(params = {}, = {}) req = build_request(:add_job_flow_steps, params) req.send_request() end |
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource, such as a cluster or an Amazon EMR Studio. Tags make it easier to associate resources in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
850 851 852 853 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 850 def (params = {}, = {}) req = build_request(:add_tags, params) req.send_request() end |
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster. Available only
in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A
maximum of 256 steps are allowed in each CancelSteps request.
CancelSteps is idempotent but asynchronous; it does not guarantee that
a step will be canceled, even if the request is successfully
submitted. When you use Amazon EMR releases 5.28.0 and later, you can
cancel steps that are in a PENDING or RUNNING state. In earlier
versions of Amazon EMR, you can only cancel steps that are in a
PENDING state.
900 901 902 903 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 900 def cancel_steps(params = {}, = {}) req = build_request(:cancel_steps, params) req.send_request() end |
#create_persistent_app_ui(params = {}) ⇒ Types::CreatePersistentAppUIOutput
Creates a persistent application user interface.
953 954 955 956 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 953 def create_persistent_app_ui(params = {}, = {}) req = build_request(:create_persistent_app_ui, params) req.send_request() end |
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
994 995 996 997 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 994 def create_security_configuration(params = {}, = {}) req = build_request(:create_security_configuration, params) req.send_request() end |
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
1125 1126 1127 1128 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1125 def create_studio(params = {}, = {}) req = build_request(:create_studio, params) req.send_request() end |
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by StudioId,
and applies a session policy to refine Studio permissions for that
user or group. Use CreateStudioSessionMapping to assign users to a
Studio when you use IAM Identity Center authentication. For
instructions on how to assign users to a Studio when you use IAM
authentication, see Assign a user or group to your EMR Studio.
1199 1200 1201 1202 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1199 def create_studio_session_mapping(params = {}, = {}) req = build_request(:create_studio_session_mapping, params) req.send_request() end |
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
1221 1222 1223 1224 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1221 def delete_security_configuration(params = {}, = {}) req = build_request(:delete_security_configuration, params) req.send_request() end |
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
1243 1244 1245 1246 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1243 def delete_studio(params = {}, = {}) req = build_request(:delete_studio, params) req.send_request() end |
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
1294 1295 1296 1297 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1294 def delete_studio_session_mapping(params = {}, = {}) req = build_request(:delete_studio_session_mapping, params) req.send_request() end |
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
The following waiters are defined for this operation (see #wait_until for detailed usage):
- cluster_running
- cluster_terminated
1416 1417 1418 1419 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1416 def describe_cluster(params = {}, = {}) req = build_request(:describe_cluster, params) req.send_request() end |
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed. We recommend you use ListClusters, DescribeCluster, ListSteps, ListInstanceGroups and ListBootstrapActions instead.
DescribeJobFlows returns a list of job flows that match all of the supplied parameters. The parameters can include a list of job flow IDs, job flow states, and restrictions on job flow creation date and time.
Regardless of supplied parameters, only job flows created within the last two months are returned.
If no parameters are supplied, then job flows matching either of the following criteria are returned:
Job flows created and completed in the last two weeks
Job flows created within the last two months that are in one of the following states:
RUNNING,WAITING,SHUTTING_DOWN,STARTING
Amazon EMR can return a maximum of 512 job flow descriptions.
1547 1548 1549 1550 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1547 def describe_job_flows(params = {}, = {}) req = build_request(:describe_job_flows, params) req.send_request() end |
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
1599 1600 1601 1602 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1599 def describe_notebook_execution(params = {}, = {}) req = build_request(:describe_notebook_execution, params) req.send_request() end |
#describe_persistent_app_ui(params = {}) ⇒ Types::DescribePersistentAppUIOutput
Describes a persistent application user interface.
1637 1638 1639 1640 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1637 def describe_persistent_app_ui(params = {}, = {}) req = build_request(:describe_persistent_app_ui, params) req.send_request() end |
#describe_release_label(params = {}) ⇒ Types::DescribeReleaseLabelOutput
Provides Amazon EMR release label details, such as the releases available the Region where the API request is run, and the available applications for a specific Amazon EMR release label. Can also list Amazon EMR releases that support a specified version of Spark.
1685 1686 1687 1688 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1685 def describe_release_label(params = {}, = {}) req = build_request(:describe_release_label, params) req.send_request() end |
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
1718 1719 1720 1721 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1718 def describe_security_configuration(params = {}, = {}) req = build_request(:describe_security_configuration, params) req.send_request() end |
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
The following waiters are defined for this operation (see #wait_until for detailed usage):
- step_complete
1778 1779 1780 1781 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1778 def describe_step(params = {}, = {}) req = build_request(:describe_step, params) req.send_request() end |
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
1830 1831 1832 1833 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1830 def describe_studio(params = {}, = {}) req = build_request(:describe_studio, params) req.send_request() end |
#get_auto_termination_policy(params = {}) ⇒ Types::GetAutoTerminationPolicyOutput
Returns the auto-termination policy for an Amazon EMR cluster.
1859 1860 1861 1862 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1859 def get_auto_termination_policy(params = {}, = {}) req = build_request(:get_auto_termination_policy, params) req.send_request() end |
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your Amazon Web Services account in the current Region. For more information see Configure Block Public Access for Amazon EMR in the Amazon EMR Management Guide.
1891 1892 1893 1894 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1891 def get_block_public_access_configuration(params = {}, = {}) req = build_request(:get_block_public_access_configuration, params) req.send_request() end |
#get_cluster_session_credentials(params = {}) ⇒ Types::GetClusterSessionCredentialsOutput
Provides temporary, HTTP basic credentials that are associated with a given runtime IAM role and used by a cluster with fine-grained access control activated. You can use these credentials to connect to cluster endpoints that support username and password authentication.
1933 1934 1935 1936 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1933 def get_cluster_session_credentials(params = {}, = {}) req = build_request(:get_cluster_session_credentials, params) req.send_request() end |
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
1968 1969 1970 1971 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 1968 def get_managed_scaling_policy(params = {}, = {}) req = build_request(:get_managed_scaling_policy, params) req.send_request() end |
#get_on_cluster_app_ui_presigned_url(params = {}) ⇒ Types::GetOnClusterAppUIPresignedURLOutput
The presigned URL properties for the cluster's application user interface.
2019 2020 2021 2022 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2019 def get_on_cluster_app_ui_presigned_url(params = {}, = {}) req = build_request(:get_on_cluster_app_ui_presigned_url, params) req.send_request() end |
#get_persistent_app_ui_presigned_url(params = {}) ⇒ Types::GetPersistentAppUIPresignedURLOutput
The presigned URL properties for the cluster's application user interface.
2069 2070 2071 2072 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2069 def get_persistent_app_ui_presigned_url(params = {}, = {}) req = build_request(:get_persistent_app_ui_presigned_url, params) req.send_request() end |
#get_session(params = {}) ⇒ Types::GetSessionOutput
Returns detailed information about a session.
2139 2140 2141 2142 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2139 def get_session(params = {}, = {}) req = build_request(:get_session, params) req.send_request() end |
#get_session_endpoint(params = {}) ⇒ Types::GetSessionEndpointOutput
Returns the Spark Connect endpoint URL and a time-limited authentication token for the specified session. Use the endpoint and token to connect a PySpark client to the session. Call this operation again when the token expires to obtain a new one.
2181 2182 2183 2184 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2181 def get_session_endpoint(params = {}, = {}) req = build_request(:get_session_endpoint, params) req.send_request() end |
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
2244 2245 2246 2247 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2244 def get_studio_session_mapping(params = {}, = {}) req = build_request(:get_studio_session_mapping, params) req.send_request() end |
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2286 2287 2288 2289 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2286 def list_bootstrap_actions(params = {}, = {}) req = build_request(:list_bootstrap_actions, params) req.send_request() end |
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this Amazon Web Services account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters in unsorted order per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2356 2357 2358 2359 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2356 def list_clusters(params = {}, = {}) req = build_request(:list_clusters, params) req.send_request() end |
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2446 2447 2448 2449 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2446 def list_instance_fleets(params = {}, = {}) req = build_request(:list_instance_fleets, params) req.send_request() end |
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2546 2547 2548 2549 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2546 def list_instance_groups(params = {}, = {}) req = build_request(:list_instance_groups, params) req.send_request() end |
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active Amazon EC2 instances and Amazon EC2 instances terminated in the last 30 days, up to a maximum of 2,000. Amazon EC2 instances in any of the following states are considered active: AWAITING_FULFILLMENT, PROVISIONING, BOOTSTRAPPING, RUNNING.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2628 2629 2630 2631 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2628 def list_instances(params = {}, = {}) req = build_request(:list_instances, params) req.send_request() end |
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions. You can filter the list
based on multiple criteria such as status, time range, and editor id.
Returns a maximum of 50 notebook executions and a marker to track the
paging of a longer notebook execution list across multiple
ListNotebookExecutions calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2724 2725 2726 2727 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2724 def list_notebook_executions(params = {}, = {}) req = build_request(:list_notebook_executions, params) req.send_request() end |
#list_release_labels(params = {}) ⇒ Types::ListReleaseLabelsOutput
Retrieves release labels of Amazon EMR services in the Region where the API is called.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2778 2779 2780 2781 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2778 def list_release_labels(params = {}, = {}) req = build_request(:list_release_labels, params) req.send_request() end |
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListSecurityConfigurations calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2816 2817 2818 2819 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2816 def list_security_configurations(params = {}, = {}) req = build_request(:list_security_configurations, params) req.send_request() end |
#list_sessions(params = {}) ⇒ Types::ListSessionsOutput
Lists the sessions on a cluster. You can filter the results by session state. Newer sessions are returned first.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2902 2903 2904 2905 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2902 def list_sessions(params = {}, = {}) req = build_request(:list_sessions, params) req.send_request() end |
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you
specify stepIds with the request or filter by StepStates. You can
specify a maximum of 10 stepIDs. The CLI automatically paginates
results to return a list greater than 50 steps. To return more than 50
steps using the CLI, specify a Marker, which is a pagination token
that indicates the next set of steps to retrieve.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2979 2980 2981 2982 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 2979 def list_steps(params = {}, = {}) req = build_request(:list_steps, params) req.send_request() end |
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the Amazon
EMR Studio specified by StudioId.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
3028 3029 3030 3031 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3028 def list_studio_session_mappings(params = {}, = {}) req = build_request(:list_studio_session_mappings, params) req.send_request() end |
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the Amazon Web Services account. The list includes details such as ID, Studio Access URL, and creation time for each Studio.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
3069 3070 3071 3072 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3069 def list_studios(params = {}, = {}) req = build_request(:list_studios, params) req.send_request() end |
#list_supported_instance_types(params = {}) ⇒ Types::ListSupportedInstanceTypesOutput
A list of the instance types that Amazon EMR supports. You can filter the list by Amazon Web Services Region and Amazon EMR release.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
3128 3129 3130 3131 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3128 def list_supported_instance_types(params = {}, = {}) req = build_request(:list_supported_instance_types, params) req.send_request() end |
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
3171 3172 3173 3174 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3171 def modify_cluster(params = {}, = {}) req = build_request(:modify_cluster, params) req.send_request() end |
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID. The call either succeeds or fails atomically.
3260 3261 3262 3263 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3260 def modify_instance_fleet(params = {}, = {}) req = build_request(:modify_instance_fleet, params) req.send_request() end |
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group. The input parameters include the new target instance count for the group and the instance group ID. The call will either succeed or fail atomically.
3315 3316 3317 3318 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3315 def modify_instance_groups(params = {}, = {}) req = build_request(:modify_instance_groups, params) req.send_request() end |
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates Amazon EC2 instances in response to the value of a CloudWatch metric.
3422 3423 3424 3425 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3422 def put_auto_scaling_policy(params = {}, = {}) req = build_request(:put_auto_scaling_policy, params) req.send_request() end |
#put_auto_termination_policy(params = {}) ⇒ Struct
Creates or updates an auto-termination policy for an Amazon EMR cluster. An auto-termination policy defines the amount of idle time in seconds after which a cluster automatically terminates. For alternative cluster termination options, see Control cluster termination.
3466 3467 3468 3469 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3466 def put_auto_termination_policy(params = {}, = {}) req = build_request(:put_auto_termination_policy, params) req.send_request() end |
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your Amazon Web Services account in the current Region. For more information see Configure Block Public Access for Amazon EMR in the Amazon EMR Management Guide.
3521 3522 3523 3524 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3521 def put_block_public_access_configuration(params = {}, = {}) req = build_request(:put_block_public_access_configuration, params) req.send_request() end |
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster. The managed scaling policy defines the limits for resources, such as Amazon EC2 instances that can be added or terminated from a cluster. The policy only applies to the core and task nodes. The master node cannot be scaled after initial configuration.
3562 3563 3564 3565 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3562 def put_managed_scaling_policy(params = {}, = {}) req = build_request(:put_managed_scaling_policy, params) req.send_request() end |
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an Amazon EMR cluster.
3591 3592 3593 3594 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3591 def remove_auto_scaling_policy(params = {}, = {}) req = build_request(:remove_auto_scaling_policy, params) req.send_request() end |
#remove_auto_termination_policy(params = {}) ⇒ Struct
Removes an auto-termination policy from an Amazon EMR cluster.
3614 3615 3616 3617 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3614 def remove_auto_termination_policy(params = {}, = {}) req = build_request(:remove_auto_termination_policy, params) req.send_request() end |
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified Amazon EMR cluster.
3637 3638 3639 3640 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3637 def remove_managed_scaling_policy(params = {}, = {}) req = build_request(:remove_managed_scaling_policy, params) req.send_request() end |
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource, such as a cluster or Amazon EMR Studio. Tags make it easier to associate resources in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
The following example removes the stack tag with value Prod from a cluster:
3679 3680 3681 3682 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 3679 def (params = {}, = {}) req = build_request(:remove_tags, params) req.send_request() end |
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow). The
cluster runs the steps specified. After the steps complete, the
cluster stops and the HDFS partition is lost. To prevent loss of data,
configure the last step of the job flow to store results in Amazon S3.
If the JobFlowInstancesConfig KeepJobFlowAliveWhenNoSteps parameter
is set to TRUE, the cluster transitions to the WAITING state rather
than shutting down after the steps have completed.
For additional protection, you can set the JobFlowInstancesConfig
TerminationProtected parameter to TRUE to lock the cluster and
prevent it from being terminated by API call, user intervention, or in
the event of a job flow error.
A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop.
For long-running clusters, we recommend that you periodically store your results.
4316 4317 4318 4319 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4316 def run_job_flow(params = {}, = {}) req = build_request(:run_job_flow, params) req.send_request() end |
#set_keep_job_flow_alive_when_no_steps(params = {}) ⇒ Struct
You can use the SetKeepJobFlowAliveWhenNoSteps to configure a
cluster (job flow) to terminate after the step execution, i.e., all
your steps are executed. If you want a transient cluster that shuts
down after the last of the current executing steps are completed, you
can configure SetKeepJobFlowAliveWhenNoSteps to false. If you want a
long running cluster, configure SetKeepJobFlowAliveWhenNoSteps to
true.
For more information, see Managing Cluster Termination in the Amazon EMR Management Guide.
4363 4364 4365 4366 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4363 def set_keep_job_flow_alive_when_no_steps(params = {}, = {}) req = build_request(:set_keep_job_flow_alive_when_no_steps, params) req.send_request() end |
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the Amazon EC2
instances in the cluster cannot be terminated by user intervention, an
API call, or in the event of a job-flow error. The cluster still
terminates upon successful completion of the job flow. Calling
SetTerminationProtection on a cluster is similar to calling the
Amazon EC2 DisableAPITermination API on all Amazon EC2 instances in
a cluster.
SetTerminationProtection is used to prevent accidental termination
of a cluster and to ensure that in the event of an error, the
instances persist so that you can recover any data stored in their
ephemeral instance storage.
To terminate a cluster that has been locked by setting
SetTerminationProtection to true, you must first unlock the job
flow by a subsequent call to SetTerminationProtection in which you
set the value to false.
For more information, see Managing Cluster Termination in the Amazon EMR Management Guide.
4416 4417 4418 4419 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4416 def set_termination_protection(params = {}, = {}) req = build_request(:set_termination_protection, params) req.send_request() end |
#set_unhealthy_node_replacement(params = {}) ⇒ Struct
Specify whether to enable unhealthy node replacement, which lets
Amazon EMR gracefully replace core nodes on a cluster if any nodes
become unhealthy. For example, a node becomes unhealthy if disk usage
is above 90%. If unhealthy node replacement is on and
TerminationProtected are off, Amazon EMR immediately terminates the
unhealthy core nodes. To use unhealthy node replacement and retain
unhealthy core nodes, use to turn on termination protection. In such
cases, Amazon EMR adds the unhealthy nodes to a denylist, reducing job
interruptions and failures.
If unhealthy node replacement is on, Amazon EMR notifies YARN and other applications on the cluster to stop scheduling tasks with these nodes, moves the data, and then terminates the nodes.
For more information, see graceful node replacement in the Amazon EMR Management Guide.
4464 4465 4466 4467 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4464 def set_unhealthy_node_replacement(params = {}, = {}) req = build_request(:set_unhealthy_node_replacement, params) req.send_request() end |
#set_visible_to_all_users(params = {}) ⇒ Struct
The SetVisibleToAllUsers parameter is no longer supported. Your cluster may be visible to all users in your account. To restrict cluster access using an IAM policy, see Identity and Access Management for Amazon EMR.
Sets the Cluster$VisibleToAllUsers value for an Amazon EMR cluster.
When true, IAM principals in the Amazon Web Services account can
perform Amazon EMR cluster actions that their IAM policies allow. When
false, only the IAM principal that created the cluster and the
Amazon Web Services account root user can perform Amazon EMR actions
on the cluster, regardless of IAM permissions policies attached to
other IAM principals.
This action works on running clusters. When you create a cluster, use the RunJobFlowInput$VisibleToAllUsers parameter.
For more information, see Understanding the Amazon EMR Cluster VisibleToAllUsers Setting in the Amazon EMR Management Guide.
4517 4518 4519 4520 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4517 def set_visible_to_all_users(params = {}, = {}) req = build_request(:set_visible_to_all_users, params) req.send_request() end |
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
4623 4624 4625 4626 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4623 def start_notebook_execution(params = {}, = {}) req = build_request(:start_notebook_execution, params) req.send_request() end |
#start_session(params = {}) ⇒ Types::StartSessionOutput
Creates and starts a new Spark Connect session on the specified
cluster. The cluster must be in the RUNNING or WAITING state and
have sessions enabled. This operation is supported in Amazon EMR Spark
8.0.0 and later.
4735 4736 4737 4738 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4735 def start_session(params = {}, = {}) req = build_request(:start_session, params) req.send_request() end |
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
4757 4758 4759 4760 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4757 def stop_notebook_execution(params = {}, = {}) req = build_request(:stop_notebook_execution, params) req.send_request() end |
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down. When a job flow is shut down, any step not yet completed is canceled and the Amazon EC2 instances on which the cluster is running are stopped. Any log files not already saved are uploaded to Amazon S3 if a LogUri was specified when the cluster was created.
The maximum number of clusters allowed is 10. The call to
TerminateJobFlows is asynchronous. Depending on the configuration of
the cluster, it may take up to 1-5 minutes for the cluster to
completely terminate and release allocated resources, such as Amazon
EC2 instances.
4789 4790 4791 4792 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4789 def terminate_job_flows(params = {}, = {}) req = build_request(:terminate_job_flows, params) req.send_request() end |
#terminate_session(params = {}) ⇒ Types::TerminateSessionOutput
Terminates an active session. After you call this operation, the
session enters the TERMINATING state and then transitions to
TERMINATED.
4827 4828 4829 4830 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4827 def terminate_session(params = {}, = {}) req = build_request(:terminate_session, params) req.send_request() end |
#update_studio(params = {}) ⇒ Struct
Updates an Amazon EMR Studio configuration, including attributes such as name, description, and subnets.
4876 4877 4878 4879 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4876 def update_studio(params = {}, = {}) req = build_request(:update_studio, params) req.send_request() end |
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
4932 4933 4934 4935 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 4932 def update_studio_session_mapping(params = {}, = {}) req = build_request(:update_studio_session_mapping, params) req.send_request() end |
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
Basic Usage
A waiter will call an API operation until:
- It is successful
- It enters a terminal state
- It makes the maximum number of attempts
In between attempts, the waiter will sleep.
# polls in a loop, sleeping between attempts
client.wait_until(waiter_name, params)
Configuration
You can configure the maximum number of polling attempts, and the delay (in seconds) between each polling attempt. You can pass configuration as the final arguments hash.
# poll for ~25 seconds
client.wait_until(waiter_name, params, {
max_attempts: 5,
delay: 5,
})
Callbacks
You can be notified before each polling attempt and before each
delay. If you throw :success or :failure from these callbacks,
it will terminate the waiter.
started_at = Time.now
client.wait_until(waiter_name, params, {
# disable max attempts
max_attempts: nil,
# poll for 1 hour, instead of a number of attempts
before_wait: -> (attempts, response) do
throw :failure if Time.now - started_at > 3600
end
})
Handling Errors
When a waiter is unsuccessful, it will raise an error. All of the failure errors extend from Waiters::Errors::WaiterFailed.
begin
client.wait_until(...)
rescue Aws::Waiters::Errors::WaiterFailed
# resource did not enter the desired state in time
end
Valid Waiters
The following table lists the valid waiter names, the operations they call,
and the default :delay and :max_attempts values.
| waiter_name | params | :delay | :max_attempts |
|---|---|---|---|
| cluster_running | #describe_cluster | 30 | 60 |
| cluster_terminated | #describe_cluster | 30 | 60 |
| step_complete | #describe_step | 30 | 60 |
5049 5050 5051 5052 5053 |
# File 'gems/aws-sdk-emr/lib/aws-sdk-emr/client.rb', line 5049 def wait_until(waiter_name, params = {}, = {}) w = waiter(waiter_name, ) yield(w.waiter) if block_given? # deprecated w.wait(params) end |