Run penetration tests from your CI/CD pipeline

You can run an AWS Security Agent penetration test as a post-deployment gate in your continuous integration and continuous delivery (CI/CD) pipeline. You add the integration as a pipeline step that runs after a deployment. The step tests the change that was just deployed and can block promotion to the next stage (for example, to production) when a finding is at or above a severity threshold that you set.

Each deploy gets security coverage automatically. You don’t have to define the test scope or wait for a scheduled assessment, because AWS Security Agent scopes each run to the change that the pipeline deployed.

AWS Security Agent provides an integration for common CI/CD platforms:

All testing runs server-side in your AWS account. The pipeline step only starts the job, waits for the result, and reports the outcome to your pipeline; no penetration testing logic runs on your CI/CD runner.

Before you start

The pipeline integration starts a job for a penetration test that you have already configured. It does not create the target or the test for you. Complete this one-time setup first:

How the pipeline integration scopes the test

When the pipeline step runs, AWS Security Agent determines the commit range to test and scopes the penetration test to that range:

You don’t normally supply commit identifiers. AWS Security Agent computes the difference server-side and validates the range, so the base that the pipeline tracks cannot be used to widen or evade the configured scope. Each integration does accept explicit base and head commit inputs for the case where you need to override the automatic range; pinning a head commit also leaves the recorded baseline unchanged, as What advances the baseline, and why that matters describes.

When the test completes, AWS Security Agent returns a scope decision that tells the pipeline how to proceed:

Every run records its decision. The Scoping tab of a run in the AWS Security Agent console shows the decision, the reasoning behind it, and the commit range that was compared:

How the pipeline integration authenticates

The pipeline integration does not use long-lived AWS access keys. Instead, it uses OpenID Connect (OIDC) federation: your CI/CD provider issues a short-lived OIDC token that identifies the pipeline, and AWS Security Token Service (AWS STS) exchanges that token for temporary credentials by having the pipeline assume an IAM role that you create in your account. The temporary credentials are used only for the duration of the pipeline step.

The security of the integration depends on two things you control:

The following sections describe how to scope both to the principle of least privilege.

Scope the trust policy to your pipeline

First, create an IAM OIDC identity provider for your CI/CD provider. The provider URL and audience differ by provider, and on Bitbucket and Azure DevOps the URL contains an identifier specific to your workspace or organization. The following section gives the value for each provider.

Then create the IAM role with a trust policy that federates that provider and restricts which repository can assume the role by matching on the token’s sub (subject) claim. Never wildcard the part of the claim that identifies the repository, project, or service connection: a sub condition such as repo:* lets any repository on that provider — including forks and repositories in other organizations — assume your role and start penetration tests billed to your account.

A trailing wildcard that covers only the ref or step portion of the claim, such as repo:my-org/my-app:*, is a different matter. The repository is still fixed, so the policy admits only that one repository. Tighten it to a specific branch or environment where your provider’s claim allows it, and note that not every provider does. On Bitbucket you cannot scope to a branch at all, and repository scope, optionally plus a deployment environment, is as tight as the claim allows.

The following example trust policy allows only the main branch of the my-org/my-app repository on GitHub Actions to assume the role. Replace the account ID, organization, repository, and branch with your own values.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:my-org/my-app:ref:refs/heads/main"
        }
      }
    }
  ]
}

The claim name and the format of the sub value differ by provider. The following table gives both for each of the four.

Grant least-privilege permissions

Attach a permissions policy that allows only the actions the pipeline step performs: starting the penetration test job, polling its status, reading findings, stopping an abandoned job during cleanup, and resolving the repository integration.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "securityagent:StartPentestJob",
        "securityagent:BatchGetPentestJobs",
        "securityagent:StopPentestJob",
        "securityagent:ListFindings",
        "securityagent:BatchGetFindings",
        "securityagent:ListIntegratedResources"
      ],
      "Resource": "*"
    }
  ]
}

Each action supports a specific part of the pipeline step:

These six actions are the complete set: the pipeline step makes no other AWS Security Agent call, and removing any one of them breaks a step that the gate performs. Do not add an sts: action to this policy. sts:GetCallerIdentity, which the credential step uses to confirm which identity it assumed, requires no permission at all, and sts:AssumeRoleWithWebIdentity is granted by the role’s trust policy rather than by what the role can do once assumed.

How the integration differs by provider

The remaining setup is specific to your CI/CD provider: the OIDC provider URL and audience, how the pipeline obtains credentials, and how the gate records the commit it last cleared. The settings you pass the gate are the same everywhere, but each provider spells their names differently.

Whichever provider you use, run the gate in a step that runs after your pre-production deployment, and gate promotion to production on its result.

Defaults the integration ships with

The settings are named slightly differently on each provider, but they map to the same concepts and share the same defaults. The defaults are identical across all four integrations.

Validate your setup before you trust the gate

Because the gate can be configured to pass the pipeline on infrastructure errors (see Choose how the pipeline responds), a misconfiguration — a bad OIDC trust policy, the wrong Region, or a repository that is not connected as an integration — can otherwise pass silently. Run the integration once in dry-run mode to confirm the setup end to end before you rely on it to gate promotion.

In dry-run mode, the integration checks AWS authentication, the Region, and that your repository resolves to its integration, then exits without starting a billable penetration test job. If validation fails, the step fails regardless of your error-handling setting, so the misconfiguration is surfaced.

Choose how the pipeline responds

Decide how the pipeline step should behave for each outcome. The following table summarizes the outcomes and the settings that control them:

Three settings fail the step independently

The three settings that decide whether the step fails are independent of each other, and each covers a different kind of outcome:

Setting the severity threshold to NONE relaxes only the first. A scope conflict still fails the step while on-scope-conflict is block, and an infrastructure error still fails it while fail-on-error is true. To make the step report without ever failing the pipeline, relax all three: set the threshold to NONE, on-scope-conflict to warn, and fail-on-error to false.

What advances the baseline, and why that matters

The gate records the commit it last cleared, and the next run tests only what changed after that commit. Any run that passes normally advances that record — including, on most providers, a run that passed without assessing anything.

That is intended. Relaxing a setting is you accepting the outcome it allows: a report-only run passes because you chose not to gate on findings, and a fail-open run passes because you chose to accept an infrastructure error. In both cases the commits in that run’s range are recorded as cleared.

The consequence is the same in both cases, and it is easy to miss: those commits are not tested again. If you later tighten the threshold, or set fail-on-error to true, the gate assesses only commits after the last recorded one. Raising the threshold does not revisit what a report-only run already accepted.

To make a blocking threshold reassess a range that an earlier run already cleared, delete the baseline record before you switch. Where it is stored differs by provider; see How the integration differs by provider.

On GitLab, the committed settings are not the gate’s contract

On GitLab CI/CD, a pipeline variable overrides the value a job sets in its own variables: block. The gate’s settings are passed that way, so if you can run a pipeline, you can supply a pipeline variable that lowers the severity threshold, turns on dry-run mode, or relaxes error handling — and the gate then does what the variable says, not what your .gitlab-ci.yml says.

The settings you commit are therefore the gate’s defaults rather than its contract, unless you also restrict who may override variables. In your project’s CI/CD settings, set the minimum role required to override variables on a pipeline to a role you trust with that decision. Restrict it before you rely on the gate to hold a release.

Report-only runs

Setting the severity threshold to NONE gives you a report-only gate: the penetration test still runs and every finding is still collected, published, and counted, but no finding fails the step. This is a useful way to see what the integration finds on real deployments before you let it block a release.

NONE relaxes findings only. A scope conflict and an infrastructure error still fail the step on their own settings, as described in Three settings fail the step independently.

Gate on scope conflicts

AWS Security Agent scopes each pipeline-triggered penetration test to the change that the pipeline deployed. When the change falls outside the penetration test’s configured scope, AWS Security Agent reports a scope conflict and does not test the change, rather than silently testing the wrong surface.

You choose how strict to be. With the pipeline integration, you can treat a scope conflict as a pipeline-blocking condition — fail the step and require someone to review the scope — or as a non-blocking signal that records the result and lets the pipeline continue. For a change that requires security sign-off, treat a scope conflict as blocking.

Cancellation and cleanup

A penetration test job runs server-side and is billable, and it keeps running until it finishes even if your pipeline stops waiting for it. When the pipeline step reaches its own timeout, the integration asks AWS Security Agent to stop the abandoned job. This cleanup uses the securityagent:StopPentestJob and securityagent:BatchGetPentestJobs permissions in the policy above, and it never fails the pipeline step.

The credentials the pipeline assumes do not refresh while the step runs, so their lifetime has to cover the whole penetration test rather than only the call that starts it. Set the IAM role’s maximum session duration to at least (timeout + 10) * 60 seconds — 4,200 seconds for the default 60-minute timeout — and, on the providers that let you request a lifetime, request that much. AWS STS rejects a request for more than the role allows rather than reducing it, so a role whose maximum is too low fails the credential exchange outright. A credential that expires partway through instead makes an otherwise healthy run fail as though the service were at fault, and it leaves the cleanup unable to stop the abandoned job.

Where you request that lifetime differs by provider: GitLab’s component derives it from the gate timeout, GitHub takes it on the credential step, and Azure DevOps takes it as a pipeline variable. On GitLab the role’s maximum session duration is one of four durations that have to agree — the gate timeout, the gate job’s own timeout, the project’s maximum job timeout, and the role’s session ceiling — and the project maximum caps the other two.

Protect credentials on the runner

The temporary AWS credentials the pipeline obtains, and the OIDC token used to obtain them, are sensitive. Anyone who can read them while the pipeline step runs can act as your pipeline. Follow these practices on your CI/CD runner:

Worked examples by provider

The following topics walk through a complete, reproducible setup for each supported CI/CD platform, using an intentionally vulnerable sample application so that you can see the gate block a real finding:

Related security topics