CreateDevEndpoint
Creates a new development endpoint.
Request Syntax
{
"Arguments": {
"string
" : "string
"
},
"EndpointName": "string
",
"ExtraJarsS3Path": "string
",
"ExtraPythonLibsS3Path": "string
",
"GlueVersion": "string
",
"NumberOfNodes": number
,
"NumberOfWorkers": number
,
"PublicKey": "string
",
"PublicKeys": [ "string
" ],
"RoleArn": "string
",
"SecurityConfiguration": "string
",
"SecurityGroupIds": [ "string
" ],
"SubnetId": "string
",
"Tags": {
"string
" : "string
"
},
"WorkerType": "string
"
}
Request Parameters
For information about the parameters that are common to all actions, see Common Parameters.
The request accepts the following data in JSON format.
- Arguments
-
A map of arguments used to configure the
DevEndpoint
.Type: String to string map
Map Entries: Minimum number of 0 items. Maximum number of 100 items.
Required: No
- EndpointName
-
The name to be assigned to the new
DevEndpoint
.Type: String
Required: Yes
- ExtraJarsS3Path
-
The path to one or more Java
.jar
files in an S3 bucket that should be loaded in yourDevEndpoint
.Type: String
Required: No
- ExtraPythonLibsS3Path
-
The paths to one or more Python libraries in an Amazon S3 bucket that should be loaded in your
DevEndpoint
. Multiple values must be complete paths separated by a comma.Note
You can only use pure Python libraries with a
DevEndpoint
. Libraries that rely on C extensions, such as the pandasPython data analysis library, are not yet supported. Type: String
Required: No
- GlueVersion
-
Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for running your ETL scripts on development endpoints.
For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.
Development endpoints that are created without specifying a Glue version default to Glue 0.9.
You can specify a version of Python support for development endpoints by using the
Arguments
parameter in theCreateDevEndpoint
orUpdateDevEndpoint
APIs. If no arguments are provided, the version defaults to Python 2.Type: String
Length Constraints: Minimum length of 1. Maximum length of 255.
Pattern:
^(\w+\.)+\w+$
Required: No
- NumberOfNodes
-
The number of AWS Glue Data Processing Units (DPUs) to allocate to this
DevEndpoint
.Type: Integer
Required: No
- NumberOfWorkers
-
The number of workers of a defined
workerType
that are allocated to the development endpoint.The maximum number of workers you can define are 299 for
G.1X
, and 149 forG.2X
.Type: Integer
Required: No
- PublicKey
-
The public key to be used by this
DevEndpoint
for authentication. This attribute is provided for backward compatibility because the recommended attribute to use is public keys.Type: String
Required: No
- PublicKeys
-
A list of public keys to be used by the development endpoints for authentication. The use of this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.
Note
If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys. Call the
UpdateDevEndpoint
API with the public key content in thedeletePublicKeys
attribute, and the list of new keys in theaddPublicKeys
attribute.Type: Array of strings
Array Members: Maximum number of 5 items.
Required: No
- RoleArn
-
The IAM role for the
DevEndpoint
.Type: String
Pattern:
arn:aws:iam::\d{12}:role/.*
Required: Yes
- SecurityConfiguration
-
The name of the
SecurityConfiguration
structure to be used with thisDevEndpoint
.Type: String
Length Constraints: Minimum length of 1. Maximum length of 255.
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*
Required: No
- SecurityGroupIds
-
Security group IDs for the security groups to be used by the new
DevEndpoint
.Type: Array of strings
Required: No
- SubnetId
-
The subnet ID for the new
DevEndpoint
to use.Type: String
Required: No
- Tags
-
The tags to use with this DevEndpoint. You may use tags to limit access to the DevEndpoint. For more information about tags in AWS Glue, see AWS Tags in AWS Glue in the developer guide.
Type: String to string map
Map Entries: Minimum number of 0 items. Maximum number of 50 items.
Key Length Constraints: Minimum length of 1. Maximum length of 128.
Value Length Constraints: Minimum length of 0. Maximum length of 256.
Required: No
- WorkerType
-
The type of predefined worker that is allocated to the development endpoint. Accepts a value of Standard, G.1X, or G.2X.
-
For the
Standard
worker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. -
For the
G.1X
worker type, each worker maps to 1 DPU (4 vCPU, 16 GB of memory, 64 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs. -
For the
G.2X
worker type, each worker maps to 2 DPU (8 vCPU, 32 GB of memory, 128 GB disk), and provides 1 executor per worker. We recommend this worker type for memory-intensive jobs.
Known issue: when a development endpoint is created with the
G.2X
WorkerType
configuration, the Spark drivers for the development endpoint will run on 4 vCPU, 16 GB of memory, and a 64 GB disk.Type: String
Valid Values:
Standard | G.1X | G.2X | G.025X | G.4X | G.8X | Z.2X
Required: No
-
Response Syntax
{
"Arguments": {
"string" : "string"
},
"AvailabilityZone": "string",
"CreatedTimestamp": number,
"EndpointName": "string",
"ExtraJarsS3Path": "string",
"ExtraPythonLibsS3Path": "string",
"FailureReason": "string",
"GlueVersion": "string",
"NumberOfNodes": number,
"NumberOfWorkers": number,
"RoleArn": "string",
"SecurityConfiguration": "string",
"SecurityGroupIds": [ "string" ],
"Status": "string",
"SubnetId": "string",
"VpcId": "string",
"WorkerType": "string",
"YarnEndpointAddress": "string",
"ZeppelinRemoteSparkInterpreterPort": number
}
Response Elements
If the action is successful, the service sends back an HTTP 200 response.
The following data is returned in JSON format by the service.
- Arguments
-
The map of arguments used to configure this
DevEndpoint
.Valid arguments are:
-
"--enable-glue-datacatalog": ""
You can specify a version of Python support for development endpoints by using the
Arguments
parameter in theCreateDevEndpoint
orUpdateDevEndpoint
APIs. If no arguments are provided, the version defaults to Python 2.Type: String to string map
Map Entries: Minimum number of 0 items. Maximum number of 100 items.
-
- AvailabilityZone
-
The AWS Availability Zone where this
DevEndpoint
is located.Type: String
- CreatedTimestamp
-
The point in time at which this
DevEndpoint
was created.Type: Timestamp
- EndpointName
-
The name assigned to the new
DevEndpoint
.Type: String
- ExtraJarsS3Path
-
Path to one or more Java
.jar
files in an S3 bucket that will be loaded in yourDevEndpoint
.Type: String
- ExtraPythonLibsS3Path
-
The paths to one or more Python libraries in an S3 bucket that will be loaded in your
DevEndpoint
.Type: String
- FailureReason
-
The reason for a current failure in this
DevEndpoint
.Type: String
- GlueVersion
-
Glue version determines the versions of Apache Spark and Python that AWS Glue supports. The Python version indicates the version supported for running your ETL scripts on development endpoints.
For more information about the available AWS Glue versions and corresponding Spark and Python versions, see Glue version in the developer guide.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 255.
Pattern:
^(\w+\.)+\w+$
- NumberOfNodes
-
The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
Type: Integer
- NumberOfWorkers
-
The number of workers of a defined
workerType
that are allocated to the development endpoint.Type: Integer
- RoleArn
-
The Amazon Resource Name (ARN) of the role assigned to the new
DevEndpoint
.Type: String
Pattern:
arn:aws:iam::\d{12}:role/.*
- SecurityConfiguration
-
The name of the
SecurityConfiguration
structure being used with thisDevEndpoint
.Type: String
Length Constraints: Minimum length of 1. Maximum length of 255.
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*
- SecurityGroupIds
-
The security groups assigned to the new
DevEndpoint
.Type: Array of strings
- Status
-
The current status of the new
DevEndpoint
.Type: String
- SubnetId
-
The subnet ID assigned to the new
DevEndpoint
.Type: String
- VpcId
-
The ID of the virtual private cloud (VPC) used by this
DevEndpoint
.Type: String
- WorkerType
-
The type of predefined worker that is allocated to the development endpoint. May be a value of Standard, G.1X, or G.2X.
Type: String
Valid Values:
Standard | G.1X | G.2X | G.025X | G.4X | G.8X | Z.2X
- YarnEndpointAddress
-
The address of the YARN endpoint used by this
DevEndpoint
.Type: String
- ZeppelinRemoteSparkInterpreterPort
-
The Apache Zeppelin port for the remote Apache Spark interpreter.
Type: Integer
Errors
For information about the errors that are common to all actions, see Common Errors.
- AccessDeniedException
-
Access to a resource was denied.
HTTP Status Code: 400
- AlreadyExistsException
-
A resource to be created or added already exists.
HTTP Status Code: 400
- IdempotentParameterMismatchException
-
The same unique identifier was associated with two different records.
HTTP Status Code: 400
- InternalServiceException
-
An internal service error occurred.
HTTP Status Code: 500
- InvalidInputException
-
The input provided was not valid.
HTTP Status Code: 400
- OperationTimeoutException
-
The operation timed out.
HTTP Status Code: 400
- ResourceNumberLimitExceededException
-
A resource numerical limit was exceeded.
HTTP Status Code: 400
- ValidationException
-
A value could not be validated.
HTTP Status Code: 400
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following: