Amazon Elastic Inference Error Codes - Amazon Elastic Inference

Amazon Elastic Inference Error Codes

The Amazon Elastic Inference service manages the lifecycle of Elastic Inference accelerators and is accessible as an AWS PrivateLink endpoint service. The client instance (Amazon Elastic Compute Cloud (Amazon EC2), Amazon SageMaker or the Amazon Elastic Container Service (Amazon ECS) container instance) connects to an AWS PrivateLink endpoint to reach an Elastic Inference accelerator. The Elastic Inference version of the framework code includes an Elastic Inference client library (ECL) that is compatible with machine learning frameworks including TensorFlow, Apache MXNet, and PyTorch. ECL communicates with the Elastic Inference accelerator through AWS PrivateLink. The Elastic Inference-enabled framework running on the client instance maintains a persistent connection to the Elastic Inference accelerator via a keep-alive thread using ECL. You can see the health status of the accelerator on your Amazon CloudWatch metrics for Elastic Inference.

When you make the first inference call to an accelerator after you provision any service instance, it takes longer than subsequent infer calls. During this time, the Elastic Inference service sets up a session between the client instance and the Elastic Inference accelerator. The client code also inspects the model’s operators. If there are any operators that cannot run on the Elastic Inference accelerator, the client code partitions the execution graph and only loads the subgraphs with operators that are supported on the accelerator. This implies that some of the subgraphs are run on the accelerator and the others are run locally. Any subsequent inference calls take less time to run because they use the already-initialized sessions. They also run on an Elastic Inference accelerator that has already loaded the model. If your model includes any operator that is not supported on Elastic Inference, the inference calls have higher latency. You can see CloudWatch metrics for the subgraphs that run on the Elastic Inference accelerator.

A list of these errors is provided in the following table. When you set up the Elastic Inference accelerators and make inference calls in the different components described, you might have errors that provide three comma-delimited numbers. For example [21, 5, 28]. Look up the third error code number (in the example, 28), which is an ECL status code, in the table here to learn more. The first two numbers are internal error codes that help Amazon investigate issues and are represented with an x and y in the following table.

[x, y, ECL STATUS CODE]

Error Description

[x,y,1]

The Elastic Inference accelerator had an error. Retry the request. If this doesn't work, upgrade to a larger Elastic Inference accelerator size.

[x,y,6]

Failed to parse the model. First, update to the latest client library version and retry. If this doesn't work, contact .

[x,y,7]

This typically happens when Elastic Inference has not been set up correctly. Use the following resources to check your Elastic Inference setup:

For SageMaker: Set Up to Use EI

For Amazon EC2: Setting Up to Launch Amazon EC2 with Elastic Inference

For Amazon ECS: Verify that your ecs-ei-task-role has been created correctly for the Amazon ECS container instance. For an example, see Setting up an ECS container instance for Elastic Inference in the blog post Running Amazon Elastic Inference Workloads on Amazon ECS.

[x,y,8]

The client instance or Amazon ECS task is unable to authenticate with the Elastic Inference accelerator. To configure the required permissions, see Configuring an Instance Role with an Elastic Inference Policy.

[x,y,9]

Authentication failed during SigV4 signing. Contact .

[x,y,10]

Stop the client instance, then start it again. If this doesn't work, provision a new client instance with a new accelerator. For Amazon ECS, stop the current task and launch a new one.

[x,y,12]

Model not loaded on the Elastic Inference accelerator. Retry your inference request. If this doesn't work, contact .

[x,y,13]

An inference session is not active for the Elastic Inference accelerator. Retry your inference request. If this doesn't work, contact .

[x,y,15]

An internal error occurred on the Elastic Inference accelerator. Retry your inference request. If this doesn't work, contact .

[x,y,16]

An internal error occurred. Retry your inference request. If this doesn't work, contact .

[x,y,17]

An internal error occurred. Retry your inference request. If this doesn't work, contact .

[x,y,19]

Typically indicates there are no accelerators attached to the Amazon EC2 or SageMaker instance. Also the client is run outside of the Amazon ECS task container. Verify your setup according to Setting Up to Launch Amazon EC2 with Elastic Inference. If this doesn't work, contact.

[x,y,23]

An internal error occurred. Contact .

[x,y,24]

An internal error occurred. Contact .

[x,y,25]

An internal error occurred. Contact .

[x,y,26]

An internal error occurred. Contact .

[x,y,28]

Configure your client instance and Elastic Inference AWS PrivateLink endpoint in the same subnet. If they already are in the same subnet, contact

[x,y,29]

An internal error occurred. Retry your inference request. If this doesn't work, contact .

[x,y,30]

Unable to connect to the Elastic Inference accelerator. Stop and restart the client instance. For Amazon ECS, stop the current task and launch a new one. If this doesn't work, contact .

[x,y,31]

Elastic Inference accelerator is out of memory. Use a larger Elastic Inference accelerator.

[x,y,32]

Tensors that are not valid were passed to the Elastic Inference accelerator. Using different input data sizes or batch sizes is not supported and might result in this error. You can either pad your data so all shapes are the same or bind the model with different shapes to use multiple executors. The latter option may result in out-of-memory errors because the model is duplicated on the accelerator.

[x,y,34]

An internal error occurred. Contact .

[x,y,35]

Unable to locate SSL certificates on the client instance. Check /etc/ssl/certs for the following certificates: ca-bundle.crt, Amazon_Root_CA_#.pem. If they are present, contact .

[x,y,36]

Your Elastic Inference accelerator is not set up properly or the Elastic Inference service is currently unavailable. First, verify that the accelerator has been set up correctly using Setting Up to Launch Amazon EC2 with Elastic Inference and retry your request after 15 seconds. If it still doesn't work, contact .

[x,y,39]

The model type that was received does not match the model type that was expected. For example, you sent an MXNet model when the accelerator was expecting a TensorFlow model. Stop and then restart the client instance to load the correct model and retry the request. For Amazon ECS, stop the current task and launch a new one with the correct model and retry the request.

[x,y,40]

 An internal error occurred. Contact .

[x,y,41]

An internal error occurred. Contact .

[x,y,42]

 Elastic Inference accelerator provisioning is in progress. Please retry your request in a few minutes.

[x,y,43]

This typically happens when the load model request took longer than the default client timeout. Retry the request to see if this resolves the issue. If it does not, contact .

[x,y,45]

 The Elastic Inference accelerator state is unknown. Stop and restart the client instance. For Amazon ECS, stop the current task and launch a new one.

[x,y,46]

 If you were unable to resolve the issue using the Elastic Inference error message provided, please contact . If applicable, please provide any Elastic Inference error codes and error messages that you received.