Troubleshooting

SAGEMAKER_TFS_INSTANCE_COUNT

For TensorFlow-based models, the tensorflow_model_server binary is the operational piece that is responsible for loading a model in memory, running inputs against a model graph, and deriving outputs. Typically, a single instance of this binary is launched to serve models in an endpoint. This binary is internally multi-threaded and spawns multiple threads to respond to an inference request. In certain instances, if you observe that the CPU is respectably utilized (over 30% utilized) but the memory is underutilized (less than 10% utilization), increasing this parameter might help. Increasing the number of tensorflow_model_servers available to serve typically increases the throughput of an endpoint.

SAGEMAKER_TFS_FRACTIONAL_GPU_MEM_MARGIN

This parameter governs the fraction of the available GPU memory to initialize CUDA/cuDNN and other GPU libraries. 0.2 means 20% of the available GPU memory is reserved for initializing CUDA/cuDNN and other GPU libraries, and 80% of the available GPU memory is allocated equally across the TF processes. GPU memory is pre-allocated unless the allow_growth option is enabled.

SAGEMAKER_TFS_INTER_OP_PARALLELISM

This ties back to the inter_op_parallelism_threads variable. This variable determines the number of threads used by independent non-blocking operations. 0 means that the system picks an appropriate number.

SAGEMAKER_TFS_INTRA_OP_PARALLELISM

This ties back to the intra_op_parallelism_threads variable. This determines the number of threads that can be used for certain operations like matrix multiplication and reductions for speedups. A value of 0 means that the system picks an appropriate number.

SAGEMAKER_GUNICORN_WORKERS

This governs the number of worker processes that Gunicorn is requested to spawn for handling requests. This value is used in combination with other parameters to derive a set that maximizes inference throughput. In addition to this, the SAGEMAKER_GUNICORN_WORKER_CLASS governs the type of workers spawned, typically async or gevent.

SAGEMAKER_GUNICORN_WORKER_CLASS

This governs the number of worker processes that Gunicorn is requested to spawn for handling requests. This value is used in combination with other parameters to derive a set that maximizes inference throughput. In addition to this, the SAGEMAKER_GUNICORN_WORKER_CLASS governs the type of workers spawned, typically async or gevent.

OMP_NUM_THREADS

Python internally uses OpenMP for implementing multithreading within processes. Typically, threads equivalent to the number of CPU cores are spawned. But when implemented on top of Simultaneous Multi Threading (SMT), such Intel’s HypeThreading, a certain process might oversubscribe a particular core by spawning twice as many threads as the number of actual CPU cores. In certain cases, a Python binary might end up spawning up to four times as many threads as available processor cores. Therefore, an ideal setting for this parameter, if you have oversubscribed available cores using worker threads, is 1, or half the number of CPU cores on a CPU with SMT turned on.

TF_DISABLE_MKL

TF_DISABLE_POOL_ALLOCATOR

In some cases, turning off MKL can speed up inference if TF_DISABLE_MKL and TF_DISABLE_POOL_ALLOCATOR are set to 1.

SAGEMAKER_TS_MAX_BATCH_DELAY

This is the maximum batch delay time TorchServe waits to receive.

SAGEMAKER_TS_BATCH_SIZE

If TorchServe doesn’t receive the number of requests specified in batch_size before the timer runs out, it sends the requests that were received to the model handler.

SAGEMAKER_TS_MIN_WORKERS

The minimum number of workers to which TorchServe is allowed to scale down.

SAGEMAKER_TS_MAX_WORKERS

The maximum number of workers to which TorchServe is allowed to scale up.

SAGEMAKER_TS_RESPONSE_TIMEOUT

The time delay, after which inference times out in absence of a response.

SAGEMAKER_TS_MAX_REQUEST_SIZE

The maximum payload size for TorchServe.

SAGEMAKER_TS_MAX_RESPONSE_SIZE

The maximum response size for TorchServe.

job_queue_size

This parameter is useful to tune when you have a scenario where the type of the inference request payload is large, and due to the size of payload being larger, you may have higher heap memory consumption of the JVM in which this queue is being maintained. Ideally you might want to keep the heap memory requirements of JVM lower and allow Python workers to allot more memory for actual model serving. JVM is only for receiving the HTTP requests, queuing them, and dispatching them to the Python-based workers for inference. If you increase the job_queue_size, you might end up increasing the heap memory consumption of the JVM and ultimately taking away memory from the host that could have been used by Python workers. Therefore, exercise caution when tuning this parameter as well.

default_workers_per_model

This parameter is for the backend model serving and might be valuable to tune since this is the critical component of the overall model serving, based on which the Python processes spawn threads for each Model. If this component is slower (or not tuned properly), the front-end tuning might not be effective.