Troubleshooting Session Manager

Use the following information to help you troubleshoot problems with AWS Systems Manager Session Manager.

Document process failed unexpectedly: document worker timed out

Problem: When starting a session to a Linux host, Systems Manager returns the following error:

document process failed unexpectedly: document worker timed out, 
check [ssm-document-worker]/[ssm-session-worker] log for crash reason

If you configured SSM Agent logging, as described in Viewing SSM Agent logs, you can view more details in the debugging log. For this issue, Session Manager shows the following log entry:

failed to create channel: too many open files

This error typically indicates that there are too many Session Manager worker processes running and the underlying operating system reached a limit. You have two options for resolving this issue.

Solution A: Increase the operating system file notification limit

You can increase the limit by running the following command from a separate Linux host. This command uses Systems Manager Run Command. The specified value increases max_user_instances to 8192. This value is considerably higher than the default value of 128, but it won't strain host resources:

aws ssm send-command --document-name AWS-RunShellScript \
--instance-id i-02573cafcfEXAMPLE  --parameters \
"commands=sudo sysctl fs.inotify.max_user_instances=8192"

Solution B: Decrease the file notifications used by Session Manager in the target host

Run the following command from a separate Linux host to list sessions running on the target host:

aws ssm describe-sessions --state Active --filters key=Target,value=i-02573cafcfEXAMPLE

Review the command output to identify sessions that are no longer needed. You can terminate those session by running the following command from a separate Linux host:

aws ssm terminate-session —session-id session ID

Optionally, once there are no more sessions running on the remote server, you can free additional resources by running the following command from a separate Linux host. This command terminates all Session Manager processes running on the remote host, and consequently all sessions to the remote host. Before you run this command, verify there are no ongoing sessions you would like to keep:

aws ssm send-command --document-name AWS-RunShellScript \
            --instance-id i-02573cafcfEXAMPLE --parameters \
'{"commands":["sudo kill $(ps aux | grep ssm-session-worker | grep -v grep | awk '"'"'{print $2}'"'"')"]}'

Session Manager can't connect from the Amazon EC2 console

Problem: After creating a new instance, the Session Manager tab in the Amazon Elastic Compute Cloud (Amazon EC2) console doesn't give you the option to connect.

Solution A: Create an instance profile: If you haven't already done so (as instructed by the information on the Session Manager tab in the EC2 console), create an AWS Identity and Access Management (IAM) instance profile by using Quick Setup. Quick Setup is a capability of AWS Systems Manager.

Session Manager requires an IAM instance profile to connect to your instance. You can create an instance profile and assign it to your instance by creating a host management configuration with Quick Setup. A host management configuration creates an instance profile with the required permissions and assigns it to your instance. A host management configuration also enables other Systems Manager capabilities and creates IAM roles for running those capabilities. There is no charge to use Quick Setup or the capabilities enabled by the host management configuration. Open Quick Setup and create a host management configuration.

If you still can't connect after creating a host management configuration, or if you receive an error, including an error about SSM Agent, see one of the following solutions:

Solution B: No error, but still can't connect

If you created the host management configuration, waited several minutes before trying to connect, and still can't connect, then you might need to manually apply the host management configuration to your instance. Use the following procedure to update a Quick Setup host management configuration and apply changes to an instance.

Wait a few minutes for EC2 to refresh the Session Manager tab. If you still can't connect or if you receive an error, review the remaining solutions for this issue.

Solution C: Error about missing SSM Agent

If you weren't able to create a host management configuration by using Quick Setup, or if you received an error about SSM Agent not being installed, you may need to manually install SSM Agent on your instance. SSM Agent is Amazon software that enables Systems Manager to connect to your instance by using Session Manager. SSM Agent is installed by default on most Amazon Machine Images (AMIs). If your instance was created from a non-standard AMI or an older AMI, you might have to manually install the agent. For the procedure to install SSM Agent, see the following topic that corresponds to your instance operating system.

For issues with SSM Agent, see Troubleshooting SSM Agent.

No permission to start a session

Problem: You try to start a session, but the system tells you that you don't have the necessary permissions.

SSM Agent not online

Problem: You see a message on the Amazon EC2 instance Session Manager tab that states: SSM Agent is not online. The SSM Agent was unable to connect to a Systems Manager endpoint to register itself with the service.

Solution: SSM Agent is Amazon software that runs on Amazon EC2 instances so that Session Manager can connect to them. If you see this error, SSM Agent is unable to establish a connection with the Systems Manager endpoint. The source of the problem could be firewall restrictions, routing problems, or lack of internet connectivity. To resolve this issue, investigate network connectivity problems.

No permission to change session preferences

Problem: You try to update global session preferences for your organization, but the system tells you that you don't have the necessary permissions.

Managed node not available or not configured for Session Manager

Problem 1: You want to start a session on the Start a session console page, but a managed node isn't in the list.

Problem 2: A managed node you want to connect is in the list on the Start a session console page, but the page reports that "The instance you selected isn't configured to use Session Manager."

Session Manager plugin not found

To use the AWS CLI to run session commands, the Session Manager plugin must also be installed on your local machine. For information, see Install the Session Manager plugin for the AWS CLI.

Session Manager plugin not automatically added to command line path (Windows)

When you install the Session Manager plugin on Windows, the session-manager-plugin executable should be automatically added to your operating system's PATH environment variable. If the command failed after you ran it to check whether the Session Manager plugin installed correctly (aws ssm start-session --target instance-id), you might need to set it manually using the following procedure.

Session Manager plugin becomes unresponsive

During a port forwarding session, traffic might stop forwarding if you have antivirus software installed on your local machine. In some cases, antivirus software interferes with the Session Manager plugin causing process deadlocks. To resolve this issue, allow or exclude the Session Manager plugin from the antivirus software. For information about the default installation path for the Session Manager plugin, see Install the Session Manager plugin for the AWS CLI.

TargetNotConnected

Problem: You try to start a session, but the system returns the error message, "An error occurred (TargetNotConnected) when calling the StartSession operation: InstanceID isn't connected."

Blank screen displays after starting a session

Problem: You start a session and Session Manager displays a blank screen.

Managed node becomes unresponsive during long running sessions

Problem: Your managed node becomes unresponsive or crashes during a long running session.

Solution: Decrease the SSM Agent log retention duration for Session Manager.

An error occurred (InvalidDocument) when calling the StartSession operation

Problem: You receive the following error when starting a session by using the AWS CLI.

An error occurred (InvalidDocument) when calling the StartSession operation: Document type: 'Command' is not supported. Only type: 'Session' is supported for Session Manager.

Solution: The SSM document you specified for the --document-name parameter isn't a Session document. Use the following procedure to view a list of Session documents in the AWS Management Console.