# Self-managed Buildkite runner in AWS CodeBuild You can configure your project to set up self-hosted Buildkite runners in CodeBuild containers to process your Buildkite jobs. This can be done by setting up a webhook using your CodeBuild project, and updating your Buildkite pipeline YAML steps to use self-hosted runners hosted on CodeBuild machines. The high-level steps to configure a CodeBuild project to run Buildkite jobs are as follows: + Navigate to the CodeBuild console and create a CodeBuild project with the Buildkite runner project runner type configuration + Add a `job.scheduled` webhook to your Buildkite organization. + Update your Buildkite pipeline YAML steps in Buildkite to configure your build environment. For a more detailed procedure, see [Tutorial: Configure a CodeBuild-hosted Buildkite runner](sample-runner-buildkite.md). This feature allows your Buildkite jobs to get native integration with AWS, which provides security and convenience through features like IAM, AWS Secrets Manager, AWS CloudTrail, and Amazon VPC. You can access the latest instance types, including ARM-based instances. # About the CodeBuild-hosted Buildkite runner The following are some common questions about the CodeBuild-hosted Buildkite runner. ## When should I include the image and instance overrides in the label? You can include the image and instance overrides in the label in order to specify different build environment for each of your Buildkite jobs. This can be done without the need to create multiple CodeBuild projects or webhooks. For example, this is useful when you need to use a [matrix for Buildkite jobs](https://buildkite.com/docs/pipelines/configure/workflows/build-matrix). ``` agents: queue: "myQueue" steps: - command: "echo \"Hello World\"" agents: project: "codebuild-myProject" image: "{{matrix.os}}" instance-size: "{{matrix.size}}" matrix: setup: os: - "arm-3.0" - "al2-5.0" size: - "small" - "large" ``` ## Can CodeBuild create webhooks within Buildkite automatically? Currently, Buildkite requires that all webhooks are created manually using their console. You can follow the tutorial at [Tutorial: Configure a CodeBuild-hosted Buildkite runner](sample-runner-buildkite.md) to create a Buildkite webhook manually in the Buildkite console. ## Can I use CloudFormation to create Buildkite webhooks? CloudFormation is not currently supported for Buildkite runner webhooks, as Buildkite requires webhooks to be created manually using their console. ## Which regions support using a CodeBuild-hosted Buildkite runner? CodeBuild-hosted Buildkite runners are supported in all CodeBuild regions. For more information about AWS Regions where CodeBuild is available, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). # Tutorial: Configure a CodeBuild-hosted Buildkite runner This tutorial shows you how to configure your CodeBuild projects to run Buildkite jobs. For more information about using Buildkite with CodeBuild see [Self-managed Buildkite runner in AWS CodeBuild](buildkite-runner.md). To complete this tutorial, you must first: + Have access to a Buildkite organization. For more information about setting up a Buildkite account and organization, you can follow this [ Getting Started Tutorial](https://buildkite.com/docs/pipelines/getting-started). + Create a Buildkite pipeline, cluster, and queue configured to use self-hosted runners. For more information about setting up these resources, you can reference the [ Buildkite Pipeline Setup Tutorial](https://buildkite.com/docs/pipelines/create-your-own). ![\[Build project in Buildkite\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-first.png) ## Step 1: Generate a Buildkite agent token In this step, you will generate an agent token within Buildkite that will be used to authenticate the CodeBuild self-hosted runners. For more information about this resource, see [Buildkite Agent Tokens](https://buildkite.com/docs/agent/v3/tokens). **To generate a Buildkite agent token** 1. In your Buildkite cluster, choose **Agent Tokens**, and then choose **New Token**. 1. Add a description to the token and click **Create Token**. 1. Save the agent token value, as it will be used later during the CodeBuild project setup. ![\[Agent tokens in Buildkite\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-createtoken.png) ## Step 2: Create a CodeBuild project with a webhook **To create a CodeBuild project with a webhook** 1. Open the AWS CodeBuild console at [https://console.aws.amazon.com/codesuite/codebuild/home](https://console.aws.amazon.com/codesuite/codebuild/home). 1. Create a self-hosted build project. For information, see [Create a build project (console)](create-project.md#create-project-console) and [Run a build (console)](run-build-console.md). + In **Project configuration**, select **Runner project**. In **Runner**: + For **Runner provider**, choose **Buildkite**. + For **Buildkite agent token**, choose **Create a new agent token by using the create secret page**. You will be prompted to create a new secret in AWS Secrets Manager with a secret value equal to the Buildkite agent token you generated above. + (Optional) If you would like to use CodeBuild managed credentials for your job, select your job’s source repository provider under **Buildkite source credential options** and verify that credentials are configured for your account. Additionally, verify that your Buildkite pipeline uses **Checkout using HTTPS**. **Note** Buildkite requires source credentials within the build environment in order to pull your job’s source. See [Authenticating Buildkite to a Private Repository](#sample-runner-buildkite-config) for available source credential options. + (Optional) In **Environment**: + Choose a supported **Environment image** and **Compute**. Note that you have the option to override the image and instance settings by using a label in your Buildkite YAML steps. For more information, see [Step 4: Update your Buildkite pipeline steps](#sample-runner-buildkite-update-pipeline). + (Optional) In **Buildspec**: + Your buildspec will be ignored by default unless `buildspec-override: "true"` is added as a label. Instead, CodeBuild will override it to use commands that will set up the self-hosted runner. **Note** CodeBuild does not support buildspec files for Buildkite self-hosted runner builds. For inline buildspecs, you will need to enable [ git-credential-helper](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec.env.git-credential-helper) in your buildspec if you have configured CodeBuild managed source credentials 1. Continue with the default values and then choose **Create build project**. 1. Save the **Payload URL** and **Secret** values from the **Create Webhook** popup. Either follow the instructions in the popup to create a new Buildkite organization webhook or continue to the next section. ## Step 3: Create a CodeBuild webhook within Buildkite In this step, you will use the **Payload URL** and **Secret** values from the CodeBuild webhook to create a new webhook within Buildkite. This webhook will be used to trigger builds within CodeBuild when a valid Buildkite job starts. **To create a new webhook in Buildkite** 1. Navigate to your Buildkite organization’s **Settings** page. 1. Under **Integrations**, select **Notification Services**. 1. Choose **Add** next to the **Webhook** box. In the **Add Webhook Notification** page, use the following configuration: 1. Under **Webhook URL**, add the saved **Payload URL** value. 1. Under **Token**, verify that **Send the token as X-Buildkite-Token** is selected. Add your webhook **Secret** value to the **Token** field. 1. Under , verify that **Send the token as X-Buildkite-Token** is selected. Add your webhook **Secret** value to the **Token** field. 1. Under **Events**, select the `job.scheduled` webhook event. 1. (Optional) Under **Pipelines**, you can optionally choose to only trigger builds for a specific pipeline. 1. Choose **Add Webhook Notification**. ## Step 4: Update your Buildkite pipeline steps In this step, you will update your Buildkite pipeline’s steps in order to add necessary labels and optional overrides. For the full list of supported label overrides, see [Label overrides supported with the CodeBuild-hosted Buildkite runner](buildkite-runner-update-labels.md). **Update your pipeline steps** 1. Navigate to your Buildkite pipeline steps page by selecting your Buildkite pipeline, choosing **Settings**, and then choosing **Steps**. If you haven’t already, choose **Convert to YAML steps**. ![\[Steps to update YAML.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-steps.png) 1. At a minimum, you will need to specify a [ Buildkite agent tag ](https://buildkite.com/docs/agent/v3/cli-start#agent-targeting) referencing the name of your CodeBuild pipeline. The project name is needed to link the AWS-related settings of your Buildkite job to a specific CodeBuild project. By including the project name in the YAML, CodeBuild is allowed to invoke jobs with the correct project settings. ``` agents: project: "codebuild-" ``` The following is an example of Buildkite pipeline steps with just the project label tag: ``` agents: project: "codebuild-myProject" steps: - command: "echo \"Hello World\"" ``` You can also override your image and compute type in the label. See [Compute images supported with the CodeBuild-hosted Buildkite runner](buildkite-runner-update-yaml.images.md) for a list of available images. The compute type and image in the label will override the environment settings on your project. To override your environment settings for a CodeBuild EC2 or Lambda compute build, use the following syntax: ``` agents: project: "codebuild-" image: "-" instance-size: "" ``` The following is an example of Buildkite pipeline steps with image and instance size overrides: ``` agents: project: "codebuild-myProject" image: "arm-3.0" instance-size: "small" steps: - command: "echo \"Hello World\"" ``` You can override the fleet used for your build in the label. This will override the fleet settings configured on your project to use the specified fleet. For more information, see [ Run builds on reserved capacity fleets ](https://docs.aws.amazon.com/codebuild/latest/userguide/fleets.html). To override your fleet settings for an Amazon EC2 compute build, use the following syntax: ``` agents: project: "codebuild-" fleet: "" ``` To override both the fleet and the image used for the build, use the following syntax: ``` agents: project: "codebuild-" fleet: "" image: "-" ``` The following is an example of Buildkite pipeline steps with fleet and image overrides: ``` agents: project: "codebuild-myProject" fleet: "myFleet" image: "arm-3.0" steps: - command: "echo \"Hello World\"" ``` 1. You can choose to run inline buildspec commands during the self-hosted Buildkite runner build (see [Run buildspec commands for the INSTALL, PRE\$1BUILD, and POST\$1BUILD phases](sample-runner-buildkite-buildspec.md) for more details). To specify that the CodeBuild build should run buildspec commands during your Buildkite self-hosted runner build, use the following syntax: ``` agents: project: "codebuild-" buildspec-override: "true" ``` The following is an example of a Buildkite pipeline with a buildspec override: ``` agents: project: "codebuild-myProject" buildspec-override: "true" steps: - command: "echo \"Hello World\"" ``` 1. Optionally, you can provide labels outside of those that CodeBuild supports. These labels will be ignored for the purpose of overriding attributes of the build, but will not fail the webhook request. For example, adding `myLabel: “testLabel"` as a label will not prevent the build from running. ## Step 5: Review your results Whenever a Buildkite job is started in your pipeline, CodeBuild will receive a `job.scheduled` webhook event through the Buildkite webhook. For each job in your Buildkite build, CodeBuild will start a build to run an ephemeral Buildkite runner. The runner is responsible for executing a single Buildkite job. Once the job is completed, the runner and the associated build process will be immediately terminated. To view your workflow job logs, navigate to your Buildkite pipeline and select the most recent build (you can trigger a new build by choosing **New Build**). Once the associated CodeBuild build for each of your jobs starts and picks up the job, you should see logs for the job within the Buildkite console ![\[Review results.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-log.png) ## Authenticating Buildkite to a Private Repository If you have a private repository configured within your Buildkite pipeline, Buildkite requires [additional permissions within the build environment](https://buildkite.com/docs/agent/v3/github-ssh-keys) to pull the repository, as Buildkite does not vend credentials to self-hosted runners to pull from private repositories. In order to authenticate the Buildkite self-hosted runner agent to your external private source repository, you can use one of the following options. **To authenticate with CodeBuild** CodeBuild offers managed credentials handling for Supported source types. In order to use CodeBuild source credentials to pull your job’s source repository, you can use the following steps: 1. In the CodeBuild console, navigate to **Edit project** or create a new CodeBuild project using the steps in [Step 2: Create a CodeBuild project with a webhook](#sample-runner-buildkite-create-project). 1. Under **Buildkite source credential options**, select your job’s source repository provider. 1. If you would like to use account-level CodeBuild credentials, verify that they are configured correctly. Additionally, if your project has an inline buildspec configured, verify that [ git-credential-helper ](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec.env.git-credential-helper) is enabled. 1. If you would like to use project level CodeBuild credentials, select **Use override credentials for this project only** and set up credentials for your project. 1. In your Buildkite pipeline settings, navigate to **Repository Settings**. Set your source repository checkout settings to **Checkout using HTTPS** ![\[Review results.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-repo-https.png) **To authenticate with Buildkite secrets** Buildkite maintains an [ ssh-checkout plugin](https://github.com/buildkite-plugins/git-ssh-checkout-buildkite-plugin) which can be used to authenticate the self-hosted runner to an external source repository using an ssh key. The key value is stored as a [Buildkite secret](https://buildkite.com/docs/pipelines/security/secrets/buildkite-secrets) and fetched automatically by the Buildkite self-hosted runner agent when attempting to pull a private repository. In order to configure the ssh-checkout plugin for your Buildkite pipeline, you can use the following steps: 1. Generate a private and public ssh key using your email address e.g. `ssh-keygen -t rsa -b 4096 -C "myEmail@address.com"` 1. Add the public key to your private source repository. For example, you can follow [this guide](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) to add a key to a GitHub account. 1. Add a [ new SSH key secret ](https://buildkite.com/docs/pipelines/hosted-agents/code-access#private-repositories-with-other-providers-add-the-ssh-key-secret) to your Buildkite cluster. Within your Buildkite cluster, select **Secrets** → **New Secret**. Add a name for you secret in the **Key** field and add your private SSH key into the **Value** field: ![\[Review results.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-secret.png) 1. Within your Buildkite pipeline, navigate to your repository settings and set checkout to use **SSH**. ![\[Review results.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-repo.png) 1. Update your pipeline YAML steps to use the `git-ssh-checkout` plugin. For example, the following pipeline YAML file uses the checkout action with the above Buildkite secret key: ``` agents: project: "codebuild-myProject" steps: - command: "npm run build" plugins: - git-ssh-checkout#v0.4.1: ssh-secret-key-name: 'SOURCE_SSH_KEY' ``` 1. When running a Buildkite self-hosted runner job within CodeBuild, Buildkite will now automatically use your configured secret value when pulling your private repository ## Runner configuration options You can specify the following environment variables in your project configuration to modify the setup configuration of your self-hosted runners: + `CODEBUILD_CONFIG_BUILDKITE_AGENT_TOKEN`: CodeBuild will fetch the secret value configured as the value of this environment variable from AWS Secrets Manager in order to register the Buildkite self-hosted runner agent. This environment variable must be of type `SECRETS_MANAGER`, and the value should be the name of your secret in Secrets Manager. A Buildkite agent token environment variable is required for all Buildkite runner projects. + `CODEBUILD_CONFIG_BUILDKITE_CREDENTIAL_DISABLE`: By default, CodeBuild will load account or project level source credentials into the build environment, as these credentials are used by the Buildkite agent to pull the job’s source repository. To disable this behavior, you can add this environment variable to your project with the value set to `true`, which will prevent source credentials from being loaded into the build environment. # Run buildspec commands for the INSTALL, PRE\$1BUILD, and POST\$1BUILD phases By default, CodeBuild ignores any buildspec commands when running a self-hosted Buildkite runner build. To run buildspec commands during the build, ``` buildspec-override: "true" ``` can be added as a suffix to the label: ``` agents: project: "codebuild-" buildspec-override: "true" ``` By using this command, CodeBuild will create a folder called `buildkite-runner` in the container's primary source folder. When the Buildkite runner starts during the `BUILD` phase, the runner will run in the `buildkite-runner` directory. There are several limitations when using a buildspec override in a self-hosted Buildkite build: + The Buildkite agent requires that source credentials exist within the build environment to pull the job’s source repository. If you use CodeBuild source credentials for authentication, you will need to enable `git-credential-helper` in your buildspec. For example, you can use the following buildspec to enable `git-credential-helper` for your Buildkite builds: ``` version: 0.2 env: git-credential-helper: yes phases: pre_build: commands: - echo "Hello World" ``` + CodeBuild will not run buildspec commands during the `BUILD` phase, as the self-hosted runner runs in the `BUILD` phase. + CodeBuild does not support buildspec files for Buildkite runner builds. Only inline buildspecs are supported for Buildlkite self-hosted runners + If a build command fails in the `PRE_BUILD` or `INSTALL` phase, CodeBuild will not start the self-hosted runner and the Buildkite job will need to be cancelled manually. # Setting up a Buildkite runner programmatically In order to configure a Buildkite runner project programatically, you will need to configure the following resources: **To create a Buildkite runner programmatically** 1. Create a Buildkite agent token and save the token in plaintext within AWS Secrets Manager. 1. Set up a CodeBuild project with your preferred configuration. You will need to configure the following additional attributes: 1. An environment value with name `CODEBUILD_CONFIG_BUILDKITE_AGENT_TOKEN`, type `SECRETS_MANAGER`, and a value equal to the Buildkite agent token associated with your Buildkite cluster. 1. Source type equal to `NO_SOURCE` 1. Permissions to access the secret created in step 1 in your project’s service role For example, you can use the following command to create a valid Buildkite runner project through the CLI: ``` aws codebuild create-project \ --name buildkite-runner-project \ --source "{\"type\": \"NO_SOURCE\",\"buildspec\":\"\"}" \ --environment "{\"image\":\"aws/codebuild/amazonlinux-x86_64-standard:5.0\",\"type\":\"LINUX_CONTAINER\",\"computeType\":\"BUILD_GENERAL1_MEDIUM\",\"environmentVariables\":[{\"name\":\"CODEBUILD_CONFIG_BUILDKITE_AGENT_TOKEN\",\"type\":\"SECRETS_MANAGER\",\"value\":\"\"}]}" \ --artifacts "{\"type\": \"NO_ARTIFACTS\"}" \ --service-role ``` 1. Create a Buildkite runner webhook on the project created in step 2. You will need to use the following configuration options when creating the webhook: 1. **build-type** should be equal to `RUNNER_BUILDKITE_BUILD` 1. A filter with type `EVENT` and a pattern equal to `WORKFLOW_JOB_QUEUED` For example, you can use the following command to create a valid Buildkite runner webhook through the CLI: ``` aws codebuild create-webhook \ --project-name buildkite-runner-project \ --filter-groups "[[{\"type\":\"EVENT\",\"pattern\":\"WORKFLOW_JOB_QUEUED\"}]]" \ --build-type RUNNER_BUILDKITE_BUILD ``` 1. Save the **Payload URL** and **Secret** values returned by the `create-webhook` call and use the credentials to create a webhook within the Buildkite console. You can reference Step 3: Create a CodeBuild webhook within Buildkite in [Tutorial: Configure a CodeBuild-hosted Buildkite runner](sample-runner-buildkite.md) for a guide on how to set up this resource. # Troubleshoot the webhook for failed builds or a hanging job **Issue: ** The webhook you set up in [Tutorial: Configure a CodeBuild-hosted Buildkite runner](sample-runner-buildkite.md) isn't working or your workflow job is hanging in Buildkite. **Possible causes: ** + Your webhook **job.scheduled** event might be failing to trigger a build. Review the **Response** logs to view the response or error message. + Your CodeBuild build fails before starting the Buildkite self-hosted runner agent to handle your job. **Recommended solutions: ** To debug failed Buildkite webhook events: 1. In your Buildkite organization settings, navigate to **Notification Services**, select your CodeBuild webhook, and then find the **Request Log**. 1. Find the `job.scheduled` webhook event associated with your stuck Buildkite job. You can use the job ID field within the webhook payload to correlate the webhook event to your Buildkite job. 1. Select the **Response** tab and check the response body. Verify that the **Response** status code is `200` and the **Response** body doesn’t contain any unexpected messages. ![\[Response for the webhook.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/buildkite-request.png) # Troubleshoot the webhook permission issues **Issue: ** The Buildkite job fails to checkout the job's source repository due to permission issues. **Possible causes: ** + CodeBuild does not have sufficient permissions to checkout the job's source repository. + The pipeline's repository settings are set to check out using SSH for CodeBuild managed credentials. **Recommended solutions: ** + Verify that CodeBuild has sufficient permissions configured to check out the job's source repository. Additionally, verify that your CodeBuild project's service role has sufficient permissions to access the configured source permission option. + Verify that your Buildkite pipeline is configured to use checkout using HTTPS if you are using CodeBuild managed source repository credentials. # Label overrides supported with the CodeBuild-hosted Buildkite runner In your Buildkite pipeline steps agent tag labels, you can provide a variety of label overrides that modify your self-hosted runner build. Any builds not recognized by CodeBuild will be ignored but will not fail your webhook request. For example, the following workflow YAML includes overrides for image, instance size, fleet, and the buildspec: ``` agents: queue: "myQueue" steps: - command: "echo \"Hello World\"" agents: project: "codebuild-myProject" image: "{{matrix.os}}" instance-size: "{{matrix.size}}" buildspec-override: "true" matrix: setup: os: - "arm-3.0" - "al2-5.0" size: - "small" - "large" ``` `project:codebuild-` (required) + Example: `project: "codebuild-myProject"` + Required for all Buildkite pipeline step configurations. ** should be equal to the name of the project for which the self-hosted runner webhook is configured. `queue: ""` + Example: `queue: ""` + Used to route Buildkite jobs to a specific queue. See the [ Buildkite Agent Queue Tag ](https://buildkite.com/docs/agent/v3/cli-start#the-queue-tag) for more information. `image: "-"` + Example: `image: "arm-3.0"` + Overrides the image and environment type used when starting the self-hosted runner build with a curated image. To learn about supported values, see [Compute images supported with the CodeBuild-hosted Buildkite runner](buildkite-runner-update-yaml.images.md). 1. To override the image and environment type used with a custom image, use `image: "custom--"` 1. Example: ``` image: "custom-arm-public.ecr.aws/codebuild/amazonlinux-aarch64-standard:3.0" ``` **Note** If the custom image resides in a private registry, you must configure the appropriate registry credentials in your CodeBuild project. `instance-size: ""` + Example: `instance-size: "medium"` + Overrides the instance type used when starting the self-hosted runner build. To learn about supported values, see [Compute images supported with the CodeBuild-hosted Buildkite runner](buildkite-runner-update-yaml.images.md). `fleet: ""` + Example: `fleet: "myFleet"` + Overrides the fleet settings configured on your project to use the specified fleet. For more information, see [ Run builds on reserved capacity fleets ](https://docs.aws.amazon.com/codebuild/latest/userguide/fleets.html). `buildspec-override: ""` + Example: `buildspec-override: "true"` + Allows the build to run buildspec commands in the `INSTALL`, `PRE_BUILD`, and `POST_BUILD` phases if set to `true`. # Compute images supported with the CodeBuild-hosted Buildkite runner In the label you configured in [Self-managed Buildkite runner in AWS CodeBuild](buildkite-runner.md), you can override your Amazon EC2 environment settings by using the values in the first three columns. CodeBuild provides the following Amazon EC2 compute images. For more information about [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/codebuild/latest/userguide/buildkite-runner-update-yaml.images.html) In addition, you can override your Lambda environment settings by using the following values. For more information about CodeBuild Lambda compute, see [Run builds on AWS Lambda compute](lambda.md). CodeBuild supports the following Lambda compute images: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/codebuild/latest/userguide/buildkite-runner-update-yaml.images.html) For more information, see [Build environment compute modes and types](build-env-ref-compute-types.md) and [Docker images provided by CodeBuild](build-env-ref-available.md).