Interface SageMakerGatewayRuntimeAsyncClient
- All Superinterfaces:
AutoCloseable,AwsClient,SdkAutoCloseable,SdkClient
builder() method.The asynchronous client performs non-blocking I/O when configured with any
SdkAsyncHttpClient supported in the SDK. However, full non-blocking is not guaranteed as the async client may
perform blocking calls in some cases such as credentials retrieval and endpoint discovery as part of the async API
call.
Use the Amazon SageMaker Gateway Runtime API to run inference through a Global Inference Gateway. A Global Inference Gateway routes each request to an inference gateway that serves the model named in the request.
The request body is an OpenAI-compatible JSON object, sent with Content-Type: application/json, and the
model is selected by the model key in that body. The content of the body is passed to the model server
unchanged, except for the stream flag, which each operation sets.
Requests are signed with Signature Version 4 under the sagemaker signing name. IAM actions are in the
sagemaker namespace, for example sagemaker:InvokeGlobalInferenceGateway. The default
regional endpoint is gateway-runtime.sagemaker.{region}.amazonaws.com.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final StringValue for looking up the service's metadata from theServiceMetadataProvider.static final String -
Method Summary
Modifier and TypeMethodDescriptionbuilder()Create a builder that can be used to configure and create aSageMakerGatewayRuntimeAsyncClient.create()Create aSageMakerGatewayRuntimeAsyncClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider.invokeGlobalInferenceGateway(Consumer<InvokeGlobalInferenceGatewayRequest.Builder> invokeGlobalInferenceGatewayRequest) Buffered (non-streaming) inference.invokeGlobalInferenceGateway(InvokeGlobalInferenceGatewayRequest invokeGlobalInferenceGatewayRequest) Buffered (non-streaming) inference.default CompletableFuture<Void> invokeGlobalInferenceGatewayWithResponseStream(Consumer<InvokeGlobalInferenceGatewayWithResponseStreamRequest.Builder> invokeGlobalInferenceGatewayWithResponseStreamRequest, InvokeGlobalInferenceGatewayWithResponseStreamResponseHandler asyncResponseHandler) Streaming inference.default CompletableFuture<Void> invokeGlobalInferenceGatewayWithResponseStream(InvokeGlobalInferenceGatewayWithResponseStreamRequest invokeGlobalInferenceGatewayWithResponseStreamRequest, InvokeGlobalInferenceGatewayWithResponseStreamResponseHandler asyncResponseHandler) Streaming inference.The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfigurationMethods inherited from interface software.amazon.awssdk.utils.SdkAutoCloseable
closeMethods inherited from interface software.amazon.awssdk.core.SdkClient
serviceName
-
Field Details
-
SERVICE_NAME
- See Also:
-
SERVICE_METADATA_ID
Value for looking up the service's metadata from theServiceMetadataProvider.- See Also:
-
-
Method Details
-
invokeGlobalInferenceGateway
default CompletableFuture<InvokeGlobalInferenceGatewayResponse> invokeGlobalInferenceGateway(InvokeGlobalInferenceGatewayRequest invokeGlobalInferenceGatewayRequest) Buffered (non-streaming) inference. Returns the full OpenAI completion body.
Errors carry their origin in the
Messageprefix. Every typed error's text is in theMessagemember (JSON keyMessage), and starts with one of these prefixes:Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>. A BadGatewayException can instead readinference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
Every error status the inference gateway returns is a typed error, chosen by that status:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
- Parameters:
invokeGlobalInferenceGatewayRequest- The request for InvokeGlobalInferenceGateway.- Returns:
- A Java Future containing the result of the InvokeGlobalInferenceGateway operation returned by the
service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
Messageprefix says whose time limit expired:Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ThrottlingException The request was throttled before it was dispatched for inference. Retry the
request with backoff, after the
Retry-Afterseconds when the response carries one. A message startinginference gatewaymeans the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted. - BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway.
The
Messageprefix says why:Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
- ServiceUnavailableException The service is temporarily unable to serve the request, for example
because no capacity is available for the requested model. The request was not dispatched for inference. A
503 returned by the inference gateway serving the model after dispatch is reported as
BadGatewayException, with the 503 in its
Message, not as this error. - InternalServerException An unexpected internal failure occurred.
- ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of
the gateway serves the model named in the request body. A model that is served but has no available
target right now returns ServiceUnavailableException instead. A message starting
inference gatewaymeans the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted. - AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the
sagemaker:InvokeGlobalInferenceGatewaypermission, orsagemaker:InvokeGlobalInferenceGatewayWithResponseStreamfor the streaming operation, on the gateway ARN. A message startinginference gatewaymeans the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted. - SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
- See Also:
-
invokeGlobalInferenceGateway
default CompletableFuture<InvokeGlobalInferenceGatewayResponse> invokeGlobalInferenceGateway(Consumer<InvokeGlobalInferenceGatewayRequest.Builder> invokeGlobalInferenceGatewayRequest) Buffered (non-streaming) inference. Returns the full OpenAI completion body.
Errors carry their origin in the
Messageprefix. Every typed error's text is in theMessagemember (JSON keyMessage), and starts with one of these prefixes:Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>. A BadGatewayException can instead readinference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
Every error status the inference gateway returns is a typed error, chosen by that status:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
This is a convenience which creates an instance of the
InvokeGlobalInferenceGatewayRequest.Builderavoiding the need to create one manually viaInvokeGlobalInferenceGatewayRequest.builder()- Parameters:
invokeGlobalInferenceGatewayRequest- AConsumerthat will call methods onInvokeGlobalInferenceGatewayRequest.Builderto create a request. The request for InvokeGlobalInferenceGateway.- Returns:
- A Java Future containing the result of the InvokeGlobalInferenceGateway operation returned by the
service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
Messageprefix says whose time limit expired:Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ThrottlingException The request was throttled before it was dispatched for inference. Retry the
request with backoff, after the
Retry-Afterseconds when the response carries one. A message startinginference gatewaymeans the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted. - BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway.
The
Messageprefix says why:Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
- ServiceUnavailableException The service is temporarily unable to serve the request, for example
because no capacity is available for the requested model. The request was not dispatched for inference. A
503 returned by the inference gateway serving the model after dispatch is reported as
BadGatewayException, with the 503 in its
Message, not as this error. - InternalServerException An unexpected internal failure occurred.
- ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of
the gateway serves the model named in the request body. A model that is served but has no available
target right now returns ServiceUnavailableException instead. A message starting
inference gatewaymeans the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted. - AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the
sagemaker:InvokeGlobalInferenceGatewaypermission, orsagemaker:InvokeGlobalInferenceGatewayWithResponseStreamfor the streaming operation, on the gateway ARN. A message startinginference gatewaymeans the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted. - SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
- See Also:
-
invokeGlobalInferenceGatewayWithResponseStream
default CompletableFuture<Void> invokeGlobalInferenceGatewayWithResponseStream(InvokeGlobalInferenceGatewayWithResponseStreamRequest invokeGlobalInferenceGatewayWithResponseStreamRequest, InvokeGlobalInferenceGatewayWithResponseStreamResponseHandler asyncResponseHandler) Streaming inference. Returns an AWS event stream of PayloadPart events that carry the model server's server-sent event (SSE) bytes, so SDKs decode the stream into events for you.
The errors listed for this operation, other than ModelStreamError and InternalStreamFailure, are returned before the response starts (before the 200 status), so they end the request, not the stream. None of them can occur after the 200. ModelStreamError and InternalStreamFailure appear only as events in the stream, after the 200. Once the 200 is sent, the stream carries the response as PayloadPart events. If reading the inference gateway's response fails, the stream carries one InternalStreamFailure event and then ends. ModelStreamError is reserved and is not currently sent.
Errors carry their origin in the
Messageprefix. Every typed error's text is in theMessagemember (JSON keyMessage), and starts with one of these prefixes:Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
An InternalStreamFailure event's
Messageuses the same prefixes, with the same text the buffered operation uses for the same failure. Itsinference gateway:message carries no status:inference gateway: response ended unexpectedlywhen the inference gateway's connection failed while it was sending the response.Every error status the inference gateway returns before the 200 is a typed error, chosen by that status as for InvokeGlobalInferenceGateway:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
- Parameters:
invokeGlobalInferenceGatewayWithResponseStreamRequest- The request for InvokeGlobalInferenceGatewayWithResponseStream. It has the same members as the InvokeGlobalInferenceGateway request.- Returns:
- A Java Future containing the result of the InvokeGlobalInferenceGatewayWithResponseStream operation
returned by the service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
Messageprefix says whose time limit expired:Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ThrottlingException The request was throttled before it was dispatched for inference. Retry the
request with backoff, after the
Retry-Afterseconds when the response carries one. A message startinginference gatewaymeans the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted. - BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway.
The
Messageprefix says why:Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ModelStreamErrorException Reserved for a model error during the stream. Not currently sent; a failure after the stream starts is reported as InternalStreamFailure.
- PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
- InternalStreamFailureException A failure after the stream started. The
Messagesays where it came from:Global Inference Gateway:when a gateway time limit expired or the inference gateway was taken out of service, orinference gateway: response ended unexpectedlywhen the inference gateway's connection failed while it was sending the response. The stream ends after this event. - ServiceUnavailableException The service is temporarily unable to serve the request, for example
because no capacity is available for the requested model. The request was not dispatched for inference. A
503 returned by the inference gateway serving the model after dispatch is reported as
BadGatewayException, with the 503 in its
Message, not as this error. - InternalServerException An unexpected internal failure occurred.
- ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of
the gateway serves the model named in the request body. A model that is served but has no available
target right now returns ServiceUnavailableException instead. A message starting
inference gatewaymeans the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted. - AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the
sagemaker:InvokeGlobalInferenceGatewaypermission, orsagemaker:InvokeGlobalInferenceGatewayWithResponseStreamfor the streaming operation, on the gateway ARN. A message startinginference gatewaymeans the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted. - SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
- See Also:
-
invokeGlobalInferenceGatewayWithResponseStream
default CompletableFuture<Void> invokeGlobalInferenceGatewayWithResponseStream(Consumer<InvokeGlobalInferenceGatewayWithResponseStreamRequest.Builder> invokeGlobalInferenceGatewayWithResponseStreamRequest, InvokeGlobalInferenceGatewayWithResponseStreamResponseHandler asyncResponseHandler) Streaming inference. Returns an AWS event stream of PayloadPart events that carry the model server's server-sent event (SSE) bytes, so SDKs decode the stream into events for you.
The errors listed for this operation, other than ModelStreamError and InternalStreamFailure, are returned before the response starts (before the 200 status), so they end the request, not the stream. None of them can occur after the 200. ModelStreamError and InternalStreamFailure appear only as events in the stream, after the 200. Once the 200 is sent, the stream carries the response as PayloadPart events. If reading the inference gateway's response fails, the stream carries one InternalStreamFailure event and then ends. ModelStreamError is reserved and is not currently sent.
Errors carry their origin in the
Messageprefix. Every typed error's text is in theMessagemember (JSON keyMessage), and starts with one of these prefixes:Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
An InternalStreamFailure event's
Messageuses the same prefixes, with the same text the buffered operation uses for the same failure. Itsinference gateway:message carries no status:inference gateway: response ended unexpectedlywhen the inference gateway's connection failed while it was sending the response.Every error status the inference gateway returns before the 200 is a typed error, chosen by that status as for InvokeGlobalInferenceGateway:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
This is a convenience which creates an instance of the
InvokeGlobalInferenceGatewayWithResponseStreamRequest.Builderavoiding the need to create one manually viaInvokeGlobalInferenceGatewayWithResponseStreamRequest.builder()- Parameters:
invokeGlobalInferenceGatewayWithResponseStreamRequest- AConsumerthat will call methods onInvokeGlobalInferenceGatewayWithResponseStreamRequest.Builderto create a request. The request for InvokeGlobalInferenceGatewayWithResponseStream. It has the same members as the InvokeGlobalInferenceGateway request.- Returns:
- A Java Future containing the result of the InvokeGlobalInferenceGatewayWithResponseStream operation
returned by the service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
Messageprefix says whose time limit expired:Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ThrottlingException The request was throttled before it was dispatched for inference. Retry the
request with backoff, after the
Retry-Afterseconds when the response carries one. A message startinginference gatewaymeans the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted. - BadGatewayException 502 -- The gateway could not get a usable response from the inference gateway.
The
Messageprefix says why:Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ModelStreamErrorException Reserved for a model error during the stream. Not currently sent; a failure after the stream starts is reported as InternalStreamFailure.
- PayloadTooLargeException The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
- InternalStreamFailureException A failure after the stream started. The
Messagesays where it came from:Global Inference Gateway:when a gateway time limit expired or the inference gateway was taken out of service, orinference gateway: response ended unexpectedlywhen the inference gateway's connection failed while it was sending the response. The stream ends after this event. - ServiceUnavailableException The service is temporarily unable to serve the request, for example
because no capacity is available for the requested model. The request was not dispatched for inference. A
503 returned by the inference gateway serving the model after dispatch is reported as
BadGatewayException, with the 503 in its
Message, not as this error. - InternalServerException An unexpected internal failure occurred.
- ResourceNotFoundException No gateway with this ID exists in this account and Region, or no target of
the gateway serves the model named in the request body. A model that is served but has no available
target right now returns ServiceUnavailableException instead. A message starting
inference gatewaymeans the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted. - AccessDeniedException The caller is not authorized to invoke this gateway. Invoking requires the
sagemaker:InvokeGlobalInferenceGatewaypermission, orsagemaker:InvokeGlobalInferenceGatewayWithResponseStreamfor the streaming operation, on the gateway ARN. A message startinginference gatewaymeans the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted. - SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerGatewayRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- GatewayTimeoutException 504 -- The inference gateway did not respond in time. The
- See Also:
-
serviceClientConfiguration
Description copied from interface:SdkClientThe SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration- Specified by:
serviceClientConfigurationin interfaceAwsClient- Specified by:
serviceClientConfigurationin interfaceSdkClient- Returns:
- SdkServiceClientConfiguration
-
create
Create aSageMakerGatewayRuntimeAsyncClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider. -
builder
Create a builder that can be used to configure and create aSageMakerGatewayRuntimeAsyncClient.
-