Frequently asked questions about cluster secret rotation in AWS PCS

A cluster secret is a secure credential that enables secure communications between the Slurm controller and AWS PCS compute nodes. It also serves as the JSON Web Token (JWT) key for Slurm REST API authentication.

In AWS PCS, the cluster secret and JWT key are the same resource serving different purposes. The cluster secret authenticates Slurm internal communications, while the JWT key signs tokens for REST API authentication. When rotated, both aspects are affected simultaneously.

The rotation process typically takes a few minutes. Your cluster remains in ACTIVE state and billing continues normally during rotation.

You can enable scheduled rotation in Secrets Manager. However, the initial release requires manual preparation (scaling node groups to 0) before each rotation.

No, existing JWT tokens become invalid after rotation. Issue new tokens for REST API clients.

You can find your cluster secret in the Secrets Manager console or through the AWS PCS console. For detailed instructions, see Use AWS Secrets Manager to find the cluster secret and Use AWS PCS to find the cluster secret.

Rotation requires no running instances to ensure cluster stability during the secret update process. This prevents authentication conflicts between old and new secrets.

This feature enables AWS PCS to meet industry compliance standards including HIPAA and FedRAMP, which mandate regular credential rotation as part of their security controls.

Yes. You can encrypt the cluster secret with a symmetric ENCRYPT_DECRYPT customer managed key in the same AWS account and AWS Region. You attach or change the key through the AWS Secrets Manager UpdateSecret operation. There is no AWS PCS API parameter for the secret's encryption key.

The key policy must grant the AWS PCS service-linked role (AWSServiceRoleForPCS) kms:Decrypt, kms:GenerateDataKey, and kms:DescribeKey. For more information, see Use a customer managed key to encrypt the cluster secret.

No. AWS PCS manages the secret value. If you try to change the SecretString or SecretBinary through AWS Secrets Manager, the request fails. You can change the encryption key (the AWS KMS key) and the description.

Rotation and node operations that read the secret fail. The underlying cause is a AWS KMS access-denied error that is visible in AWS CloudTrail, and the rotation surfaces as a failed rotation. The cluster itself remains ACTIVE.