Interface SageMakerGatewayRuntimeClient
- All Superinterfaces:
AutoCloseable,AwsClient,SdkAutoCloseable,SdkClient
builder() method.
Use the Amazon SageMaker Gateway Runtime API to run inference through a Global Inference Gateway. A Global Inference Gateway routes each request to an inference gateway that serves the model named in the request.
The request body is an OpenAI-compatible JSON object, sent with Content-Type: application/json, and the
model is selected by the model key in that body. The content of the body is passed to the model server
unchanged, except for the stream flag, which each operation sets.
Requests are signed with Signature Version 4 under the sagemaker signing name. IAM actions are in the
sagemaker namespace, for example sagemaker:InvokeGlobalInferenceGateway. The default
regional endpoint is gateway-runtime.sagemaker.{region}.amazonaws.com.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final StringValue for looking up the service's metadata from theServiceMetadataProvider.static final String -
Method Summary
Modifier and TypeMethodDescriptionbuilder()Create a builder that can be used to configure and create aSageMakerGatewayRuntimeClient.create()Create aSageMakerGatewayRuntimeClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider.invokeGlobalInferenceGateway(Consumer<InvokeGlobalInferenceGatewayRequest.Builder> invokeGlobalInferenceGatewayRequest) Buffered (non-streaming) inference.invokeGlobalInferenceGateway(InvokeGlobalInferenceGatewayRequest invokeGlobalInferenceGatewayRequest) Buffered (non-streaming) inference.The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfigurationstatic ServiceMetadataMethods inherited from interface software.amazon.awssdk.utils.SdkAutoCloseable
closeMethods inherited from interface software.amazon.awssdk.core.SdkClient
serviceName
-
Field Details
-
SERVICE_NAME
- See Also:
-
SERVICE_METADATA_ID
Value for looking up the service's metadata from theServiceMetadataProvider.- See Also:
-
-
Method Details
-
invokeGlobalInferenceGateway
default InvokeGlobalInferenceGatewayResponse invokeGlobalInferenceGateway(InvokeGlobalInferenceGatewayRequest invokeGlobalInferenceGatewayRequest) throws GatewayTimeoutException, ThrottlingException, BadGatewayException, PayloadTooLargeException, ServiceUnavailableException, InternalServerException, ResourceNotFoundException, AccessDeniedException, AwsServiceException, SdkClientException, SageMakerGatewayRuntimeException Buffered (non-streaming) inference. Returns the full OpenAI completion body.
Errors carry their origin in the
Messageprefix. Every typed error's text is in theMessagemember (JSON keyMessage), and starts with one of these prefixes:Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>. A BadGatewayException can instead readinference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
Every error status the inference gateway returns is a typed error, chosen by that status:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
- Parameters:
invokeGlobalInferenceGatewayRequest- The request for InvokeGlobalInferenceGateway.- Returns:
- Result of the InvokeGlobalInferenceGateway operation returned by the service.
- See Also:
-
invokeGlobalInferenceGateway
default InvokeGlobalInferenceGatewayResponse invokeGlobalInferenceGateway(Consumer<InvokeGlobalInferenceGatewayRequest.Builder> invokeGlobalInferenceGatewayRequest) throws GatewayTimeoutException, ThrottlingException, BadGatewayException, PayloadTooLargeException, ServiceUnavailableException, InternalServerException, ResourceNotFoundException, AccessDeniedException, AwsServiceException, SdkClientException, SageMakerGatewayRuntimeException Buffered (non-streaming) inference. Returns the full OpenAI completion body.
Errors carry their origin in the
Messageprefix. Every typed error's text is in theMessagemember (JSON keyMessage), and starts with one of these prefixes:Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>. A BadGatewayException can instead readinference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
Every error status the inference gateway returns is a typed error, chosen by that status:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
This is a convenience which creates an instance of the
InvokeGlobalInferenceGatewayRequest.Builderavoiding the need to create one manually viaInvokeGlobalInferenceGatewayRequest.builder()- Parameters:
invokeGlobalInferenceGatewayRequest- AConsumerthat will call methods onInvokeGlobalInferenceGatewayRequest.Builderto create a request. The request for InvokeGlobalInferenceGateway.- Returns:
- Result of the InvokeGlobalInferenceGateway operation returned by the service.
- See Also:
-
create
Create aSageMakerGatewayRuntimeClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider. -
builder
Create a builder that can be used to configure and create aSageMakerGatewayRuntimeClient. -
serviceMetadata
-
serviceClientConfiguration
Description copied from interface:SdkClientThe SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration- Specified by:
serviceClientConfigurationin interfaceAwsClient- Specified by:
serviceClientConfigurationin interfaceSdkClient- Returns:
- SdkServiceClientConfiguration
-