Worker host setup and configuration - AWS Deadline Cloud
Configure a Python environmentInstall worker agentConfigure worker agentCreate job users and groups

Worker host setup and configuration

A worker host refers to a host machine that runs a Deadline Cloud worker. This section explains how to set up the worker host and configure it for your specific needs. Each worker host runs a program called a worker agent. The worker agent is responsible for:

  • Managing the worker life cycle.

  • Synchronizing assigned work, its progress and results.

  • Monitoring running work.

  • Forwarding logs to configured destinations.

We recommend that you use the provided Deadline Cloud worker agent. The worker agent is open source and we encourage feature requests, but you can also develop and customize to fit your needs.

To complete the tasks in the following sections, you need the following:

Linux

  • A Linux-based Amazon Elastic Compute Cloud (Amazon EC2) instance. We recommend Amazon Linux 2023.

  • sudo privileges.

  • Python 3.9 or above.

Windows

  • A Windows-based Amazon Elastic Compute Cloud (Amazon EC2) instance. We recommend Windows Server 2022.

  • Administrator access to the worker host

  • Python 3.9 or above installed for all users

Create and configure a Python virtual environment

You can create a Python virtual environment on Linux if you have installed Python 3.9 or greater and placed it in your PATH.

Note

On Windows, agent files must be installed into Python’s global site-packages directory. Python virtual environments are not currently supported.

To create and activate a Python virtual environment

  1. Open the AWS CLI.

  2. Create and activate a Python virtual environment.

    python3 -m venv /opt/deadline/worker
source /opt/deadline/worker/bin/activate
pip install --upgrade pip

Install Deadline Cloud worker agent

After you've set up your Python and created a virtual environment on Linux, install the Deadline Cloud worker agent Python packages.

To install the worker agent Python packages

  1. Open a terminal.

    1. On Linux, open a terminal as the root user (or use sudo / su)

    2. On Windows, open an administrator command prompt or PowerShell terminal.

  2. Download and install the Deadline Cloud worker agent packages from PyPI:

    python -m pip install deadline-cloud-worker-agent

Configure the Deadline Cloud worker agent

You can configure the Deadline Cloud worker agent settings in three ways. We recommend your use the operating system set up using the install-deadline-worker tool.

The worker agent does not support running as a domain user on Windows. To run a job as a domain user, you can specify a domain user account when you configure a queue user for running jobs. For more information, see step 7 in Create a queue.

Command line arguments — You can specify arguments when you run the Deadline Cloud worker agent from the command line. Some configuration settings are not available through command line arguments. To see all the available command line arguments, enter deadline-worker-agent --help to see all the available command line arguments.

Environment variables — You can configure the Deadline Cloud worker agent by setting environment variable beginning with DEADLINE_WORKER_. For example, you can use export DEADLINE_WORKER_VERBOSE=true to set the worker agent's output to verbose. For more examples and information, see /etc/amazon/deadline/worker.toml.example on Linux or C:\ProgramData\Amazon\Deadline\Config\worker.toml.example on Windows.

Configuration file — When you install the worker agent, it creates a configuration file located at /etc/amazon/deadline/worker.toml on Linux or C:\ProgramData\Amazon\Deadline\Config\worker.toml on Windows. The worker agent loads this configuration file when it starts. You can use the example configuration file (/etc/amazon/deadline/worker.toml.example on Linux or C:\ProgramData\Amazon\Deadline\Config\worker.toml.example on Windows) to tailor the default worker agent configuration file for your specific needs.

Finally, we recommend you enable auto shutdown for the worker agent after your software is deployed and working as expected. This allows the worker fleet to scale up when needed and to shut down when the rendering job finishes. Auto scaling helps ensure you're only using resources as needed. To enable a instance started by the auto scaling group to shut down, you must add shutdown_on_stop=true to the worker.toml configuration file.

To enable auto shutdown

As a root user:

  • Install the worker agent with parameters --allow-shutdown.

    Linux

    Enter:

    /opt/deadline/worker/bin/install-deadline-worker \
  --farm-id FARM_ID \
  --fleet-id FLEET_ID \
  --region REGION \
  --allow-shutdown
    Windows

    Enter:

    install-deadline-worker ^
  --farm-id FARM_ID ^
  --fleet-id FLEET_ID ^
  --region REGION ^
  --allow-shutdown

Create job users and groups

This section describes the required user and group relationship between the agent user and the jobRunAsUser defined on your queues.

The Deadline Cloud worker agent should run as a dedicated agent-specific user on the host. You should configure the jobRunAsUser property of Deadline Cloud queues so that workers will run the queue jobs as a specific operating system user and group. This means you can control the shared filesystem permissions that your jobs have. It also provides as an important security boundary between your jobs and the worker agent user.

Linux job users and groups

To set up your agent-user and jobRunAsUser, ensure you meet the following requirements:

  • There is a group for each jobRunAsUser, and it is the primary group for its corresponding jobRunAsUser.

  • The agent-user belongs to the primary group of the jobRunAsUser for the queues where the worker obtains work. For security best-practices, we recommend this as a secondary group of the agent-user. This shared group allows the worker agent to make files available for the job while it is running.

  • A jobRunAsUser doesn't belong to the agent-user's primary group. For security best-practices:

    • Sensitive files written by the worker agent are owned by the agent's primary group.

    • If a jobRunAsUser belongs to this group, and files the worker agent writes may be accessible by the jobs submitted to the queue running on the worker.

  • The default AWS Region should match the Region of the farm the worker belongs to. For more information, see Configuration and credential file settings.

    This should be applied to:

    • The agent-user

    • All queue jobRunAsUser accounts on the worker

  • The agent-user can run sudo commands as the jobRunAsUser.

The following diagram illustrates the relationship between the agent user and the jobRunAsUser users and groups for queues associated with the fleet.

An illustration of the relationship between agent-users and the jobRunAsUser on queues.

Windows users

To use a Windows user as the jobRunAsUser, it must meet the following requirements:

  • All queue jobRunAsUser users must exist.

  • Their passwords must match the value of the secret specified in their queue’s JobRunAsUser field. For instructions, see step 7 in Create a queue.

  • The agent-user must be able to log on as those users.