Direct SSH

Before enabling, review the following:

Prerequisites

Direct SSH requires ExternalDNS, an Amazon Route 53 Private Hosted Zone, VPC connectivity from client machines, and (optionally) the AWS Load Balancer Controller. For the full list of prerequisites, see Prerequisites for Direct SSH.

If web browser access is already enabled on your cluster, all prerequisites are in place. Before you proceed, verify your ExternalDNS is configured with --policy=sync. For details, see ExternalDNS configuration.

If web browser access is not yet configured, see (Optional) Setting up prerequisites. For more information about enabling web browser access, see EKS Addon Installation - Jupyter K8s with WebUI.

Configure Direct SSH for your cluster

Step 1: Add security group rule for SSH

Web browser access uses HTTPS (443) through an Application Load Balancer (ALB). Because Direct SSH connects directly to pod IPs on the configured port, you need to add a security group inbound rule.

Check whether your workspace instance group has an SG override:

aws sagemaker describe-cluster \
  --cluster-name <HYPERPOD_CLUSTER_NAME> \
  --region <AWS_REGION> \
  --query 'InstanceGroups[*].{Name:InstanceGroupName, OverrideSGs:OverrideVpcConfig.SecurityGroupIds}'

Option A: The instance group has a security group override

If OverrideSGs is non-null, add the rule to that Security Group:

SG_ID=<SG_ID_FROM_OVERRIDE_VPC_CONFIG>
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port <SSH_PORT> \
  --cidr <SOURCE_CIDR> \
  --region <AWS_REGION>

Option B: The instance group has no security group override

If OverrideSGs is null, use the cluster-level Security Group from the SageMaker HyperPod cluster VPC configuration:

SG_ID=$(aws sagemaker describe-cluster --cluster-name <HYPERPOD_CLUSTER_NAME> --region <AWS_REGION> \
  --query 'VpcConfig.SecurityGroupIds[0]' --output text)
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port <SSH_PORT> \
  --cidr <SOURCE_CIDR> \
  --region <AWS_REGION>

The following table shows where to find each placeholder value.

Step 2: Enable Direct SSH

Add directSSH to your addon configuration alongside clusterWebUI:

jupyter-k8s-aws-hyperpod:
  clusterWebUI:
    enabled: true
    domain: "<DOMAIN_NAME>"
    awsCertificateArn: "<ACM_CERTIFICATE_ARN>"
    traefik:
      shouldInstall: true
  directSSH:
    enabled: true
    port: <SSH_PORT>          # default: 22
    domain: "<ROUTE53_HOSTED_ZONE_DOMAIN>"

The following table shows where to find each placeholder value.

Update the addon:

aws eks update-addon \
  --cluster-name <CLUSTER_NAME> \
  --addon-name amazon-sagemaker-spaces \
  --configuration-values file://addon-config.yaml \
  --resolve-conflicts OVERWRITE \
  --region <AWS_REGION>

To disable Direct SSH, see Revoking Direct SSH access.

Step 3: Verify Direct SSH is active

# Check addon status
aws eks describe-addon \
  --cluster-name <CLUSTER_NAME> \
  --addon-name amazon-sagemaker-spaces \
  --region <AWS_REGION>

# Check headless services created for workspaces
kubectl get svc -l app.kubernetes.io/component=direct-ssh

# Verify DNS record resolves (from within VPC)
dig <workspace-name>.<namespace>.<ROUTE53_HOSTED_ZONE_DOMAIN>

Connect to your workspace using Direct SSH

Use the following procedures to connect to your workspace using Direct SSH, manage your SSH keys, and troubleshoot connection problems.

How SSH authentication works

Direct SSH uses standard public key authentication. Each user generates their own SSH key pair and adds the public key to their workspace. Private keys never leave the user's machine.

The following are key facts about SSH access:

Step 1: Generate an SSH key on your client machine

Run this command once to create your key pair:

ssh-keygen -t ed25519 -f ~/.ssh/my-workspace-key

This creates:

Step 2: Add your public key to your workspace

Use either of the following methods to add your public key. You do not need to do both.

Add your key through web browser access

Add your key through kubectl

POD=$(kubectl get pods -n <namespace> -l workspace.jupyter.org/workspace-name=<space-name> \
  -o jsonpath='{.items[0].metadata.name}')
cat ~/.ssh/my-workspace-key.pub | kubectl exec -n <namespace> -i $POD -c workspace -- \
  bash -c "cat >> /home/sagemaker-user/.ssh/authorized_keys"

Step 3: Connect to your workspace using SSH

ssh -p <SSH_PORT> -i ~/.ssh/my-workspace-key sagemaker-user@<space-name>.<namespace>.<domain>

Example:

ssh -p 2222 -i ~/.ssh/my-workspace-key sagemaker-user@my-space.default.spaces.example.com

Step 4: Configure SSH for easy access

This optional step simplifies future connections. Add the following to ~/.ssh/config:

Host my-space
    HostName <space-name>.<namespace>.<domain>
    Port <SSH_PORT>
    User sagemaker-user
    IdentityFile ~/.ssh/my-workspace-key
    ServerAliveInterval 15
    ServerAliveCountMax 3

Set Port to the port your administrator configured. You can omit this line if Direct SSH uses the default port (22). Then connect with: ssh my-space

Step 5: Connect a remote IDE

After Direct SSH works from your terminal (Step 3), any Remote-SSH capable IDE connects using the same ~/.ssh/config entry. You do not need additional AWS configuration for this connection.

Install the Remote-SSH extension

Connect from the IDE

Key management and revocation

You manage your SSH keys. SSH keys are long-lived credentials with no automatic expiry. A key in authorized_keys grants access to your workspace until you explicitly remove it.

To rotate your key

We recommend rotating your keys periodically.

If your private key is compromised

Act immediately. A compromised key grants full workspace access until you remove it from authorized_keys.

Best practices

Revoking Direct SSH access

To avoid leaving an open security group rule after you disable Direct SSH, complete all three steps in order. Do not skip Step 2.

Step 1: Disable in Helm chart

Remove the directSSH section from your addon configuration file entirely. Helm defaults directSSH.enabled to false when the key is absent:

jupyter-k8s-aws-hyperpod:
  clusterWebUI:
    enabled: true
    domain: "<DOMAIN_NAME>"
    ...
  # directSSH section removed

Alternatively, explicitly set enabled: false:

directSSH:
  enabled: false

Step 2: Remove the security group inbound rule

# Find the rule ID
aws ec2 describe-security-group-rules \
  --filters Name=group-id,Values=<SG_ID> \
  --query 'SecurityGroupRules[?IpProtocol==`tcp` && FromPort==`<SSH_PORT>`].[SecurityGroupRuleId,CidrIpv4]' \
  --output table \
  --region <AWS_REGION>

# Remove the rule
aws ec2 revoke-security-group-ingress \
  --group-id <SG_ID> \
  --security-group-rule-ids <RULE_ID> \
  --region <AWS_REGION>

Step 3: Apply to cluster

aws eks update-addon \
  --cluster-name <CLUSTER_NAME> \
  --addon-name amazon-sagemaker-spaces \
  --configuration-values file://addon-config.yaml \
  --resolve-conflicts OVERWRITE \
  --region <AWS_REGION>

Running this command removes the headless Services for all workspaces. ExternalDNS then deletes the Route 53 A records within approximately 30 seconds. After ExternalDNS removes the records, new SSH connections can no longer resolve the workspace hostname.

What happens after revocation

Troubleshooting

(Optional) Setting up prerequisites

Use this section only if web browser access is not already enabled, or to verify or update your ExternalDNS configuration.

Installing from scratch

If web browser access is not yet configured, you need the following before enabling Direct SSH:

For the additional dependencies and the web browser access configuration steps, see Install SageMaker AI Spaces Add-on.

AWS Load Balancer Controller: HyperPod vpcId requirement

The standard AWS Load Balancer Controller installation documentation does not mention the vpcId parameter. On HyperPod clusters, omitting vpcId causes the installation to fail. You must provide it explicitly.

Get your VPC ID:

aws sagemaker describe-cluster \
  --cluster-name <HYPERPOD_CLUSTER_NAME> \
  --region <AWS_REGION> \
  --query 'VpcConfig.VpcId' \
  --output text

Install with the required HyperPod parameters:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=<EKS_CLUSTER_NAME> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set enableServiceMutatorWebhook=false \
  --set vpcId=<VPC_ID>

Create the service account (aws-load-balancer-controller) with the appropriate IAM role annotation before running this command. For more information about the IAM policy and service account creation steps, see Amazon EKS Load Balancer Controller IRSA setup in the Amazon EKS User Guide.

ExternalDNS configuration

ExternalDNS synchronizes Kubernetes Services with DNS providers. Configure ExternalDNS with the following settings when you use it with Amazon Route 53 in a production environment.

By default, ExternalDNS uses --policy=upsert-only. This creates and updates DNS records but never deletes them. When you delete a workspace, the A and TXT records in Amazon Route 53 remain as stale entries.

Use --policy=upsert-only only for testing, and change it to --policy=sync for production. You must also set the --txt-owner-id flag, which tells ExternalDNS which records it owns and which records to delete on cleanup.

Configure your ExternalDNS deployment with the following arguments:

--provider=aws
--source=service                          # watches Services (required for headless Services)
--domain-filter=<ROUTE53_HOSTED_ZONE>     # restricts ExternalDNS to your hosted zone only
--policy=sync                             # enables deletion of stale records on space deletion
--txt-owner-id=<CLUSTER_NAME>             # identifies which records this ExternalDNS instance owns

In the preceding arguments, replace the following values:

To verify your current ExternalDNS policy, run the following command:

kubectl get deployment -n kube-system external-dns \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -E 'policy|owner|source|domain'