# Cache builds to improve performance You can save time when your project builds by using a cache. A cache can store reusable pieces of your build environment and use them across multiple builds. Your build project can use one of two types of caching: Amazon S3 or local. If you use a local cache, you must choose one or more of three cache modes: source cache, Docker layer cache, and custom cache. **Note** Docker layer cache mode is available for the Linux environment only. If you choose this mode, you must run your build in privileged mode. CodeBuild projects granted privileged mode grants its container access to all devices. For more information, see [Runtime privilege and Linux capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) on the Docker Docs website. **Topics** + [Amazon S3 caching](caching-s3.md) + [Local caching](caching-local.md) + [Specify a local cache](specify-caching-local.md) # Amazon S3 caching Amazon S3 caching stores the cache in an Amazon S3 bucket that is available across multiple build hosts. This is a good option for small to intermediate sized build artifacts that are more expensive to build than to download. To use Amazon S3 in a build, you can specify the paths for the files you want to cache in `buildspec.yml`. CodeBuild will automatically store and update the cache to the Amazon S3 location configured on the project. If you don't specify the file paths, CodeBuild will best-effort cache common language dependencies to help you speed up the builds. You can view the cache details in the build logs. Additionally, if you want to have multiple versions of cache, you can define a cache key in the `buildspec.yml`. CodeBuild stores the cache under the context of this cache key, and create a unique cache copy that will not be updated once created. The cache keys can be shared across projects as well. Features such as dynamic keys, cache versioning, and cache sharing across builds are only available when a key is specified. To learn more about the cache syntax in buildspec file, see [cache](build-spec-ref.md#build-spec.cache) in the buildspec reference. **Topics** + [Generate dynamic keys](#caching-s3-dynamic) + [codebuild-hash-files](#caching-s3-dynamic.codebuild-hash-files) + [Cache version](#caching-s3-version) + [Cache sharing between projects](#caching-s3-sharing) + [Buildspec examples](#caching-s3-examples) ## Generate dynamic keys A cache key can include shell commands and environment variables to make it unique, enabling automatic cache updates when key changes. For example, you can define a key using the hash of the `package-lock.json` file. When the dependencies in that file change, the hash—and therefore the cache key—changes, triggering the automatic creation of a new cache. ``` cache: key: npm-key-$(codebuild-hash-files package-lock.json) ``` CodeBuild will evaluate the expression `$(codebuild-hash-files package-lock.json)` to get the final key: ``` npm-key-abc123 ``` You can also define a cache key using environment variables, such as `CODEBUILD_RESOLVED_SOURCE_VERSION`. This ensures that whenever your source changes, a new key is generated, resulting in a new cache being saved automatically: ``` cache: key: npm-key-$CODEBUILD_RESOLVED_SOURCE_VERSION ``` CodeBuild will evaluate the expression and get the final key: ``` npm-key-046e8b67481d53bdc86c3f6affdd5d1afae6d369 ``` ## codebuild-hash-files `codebuild-hash-files` is a CLI tool that calculates a SHA-256 hash for a set of files in the CodeBuild source directory using glob patterns: ``` codebuild-hash-files ... ``` Here are some examples using `codebuild-hash-files`: ``` codebuild-hash-files package-lock.json codebuild-hash-files '**/*.md' ``` ## Cache version The cache version is a hash generated from the paths of the directories being cached. If two caches have different versions, they are treated as distinct caches during the matching process. For example, the following two caches are considered different because they reference different paths: ``` version: 0.2 phases: build: commands: - pip install pandas==2.2.3 --target pip-dependencies cache: key: pip-dependencies paths: - "pip-dependencies/**/*" ``` ``` version: 0.2 phases: build: commands: - pip install pandas==2.2.3 --target tmp/pip-dependencies cache: key: pip-dependencies paths: - "tmp/pip-dependencies/**/*" ``` ## Cache sharing between projects You can use the `cacheNamespace` API field under the `cache` section to share a cache across multiple projects. This field defines the scope of the cache. To share a cache, must do the following: + Use the same `cacheNamespace`. + Specify the same cache `key`. + Define identical cache paths. + Use the same Amazon S3 buckets and `pathPrefix` if set. This ensures consistency and enables cache sharing across projects. ### Specify a cache namespace (console) 1. Open the AWS CodeBuild console at [https://console.aws.amazon.com/codesuite/codebuild/home](https://console.aws.amazon.com/codesuite/codebuild/home). 1. Choose **Create project**. For information, see [Create a build project (console)](create-project.md#create-project-console) and [Run a build (console)](run-build-console.md). 1. In **Artifacts**, choose **Additional configuration**. 1. For **Cache type**, choose **Amazon S3**. 1. For **Cache namespace - optional**, enter a namespace value. ![\[Cache namespace parameter in the CodeBuild console.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/s3-cache-namespace.png) 1. Continue with the default values and then choose **Create build project**. ### Specify a cache namespace (AWS CLI) You can use the the `--cache` parameter in the AWS CLI to specify a cache namespace. ``` --cache '{"type": "S3", "location": "your-s3-bucket", "cacheNamespace": "test-cache-namespace"}' ``` ## Buildspec examples Here are several buildspec examples for common languages: **Topics** + [Cache Node.js dependencies](#caching-s3-examples.nodejs) + [Cache Python dependencies](#caching-s3-examples.python) + [Cache Ruby dependencies](#caching-s3-examples.ruby) + [Cache Go dependencies](#caching-s3-examples.go) ### Cache Node.js dependencies If your project includes a `package-lock.json` file and uses `npm` to manage Node.js dependencies, the following example shows how to set up caching. By default, `npm` installs dependencies into the `node_modules` directory. ``` version: 0.2 phases: build: commands: - npm install cache: key: npm-$(codebuild-hash-files package-lock.json) paths: - "node_modules/**/*" ``` ### Cache Python dependencies If your project includes a `requirements.txt` file and uses pip to manage Python dependencies, the following example demonstrates how to configure caching. By default, pip installs packages into the system's `site-packages` directory. ``` version: 0.2 phases: build: commands: - pip install -r requirements.txt cache: key: python-$(codebuild-hash-files requirements.txt) paths: - "/root/.pyenv/versions/${python_version}/lib/python${python_major_version}/site-packages/**/*" ``` Additionally, you can install dependencies into a specific directory and configure caching for that directory. ``` version: 0.2 phases: build: commands: - pip install -r requirements.txt --target python-dependencies cache: key: python-$(codebuild-hash-files requirements.txt) paths: - "python-dependencies/**/*" ``` ### Cache Ruby dependencies If your project includes a `Gemfile.lock` file and uses `Bundler` to manage gem dependencies, the following example demonstrates how to configure caching effectively. ``` version: 0.2 phases: build: commands: - bundle install --path vendor/bundle cache: key: ruby-$(codebuild-hash-files Gemfile.lock) paths: - "vendor/bundle/**/*" ``` ### Cache Go dependencies If your project includes a `go.sum` file and uses Go modules to manage dependencies, the following example demonstrates how to configure caching. By default, Go modules are downloaded and stored in the `${GOPATH}/pkg/mod` directory. ``` version: 0.2 phases: build: commands: - go mod download cache: key: go-$(codebuild-hash-files go.sum) paths: - "/go/pkg/mod/**/*" ``` # Local caching Local caching stores a cache locally on a build host that is available to that build host only. This is a good option for intermediate to large build artifacts because the cache is immediately available on the build host. This is not the best option if your builds are infrequent. This means that build performance is not impacted by network transfer time. If you choose local caching, you must choose one or more of the following cache modes: + Source cache mode caches Git metadata for primary and secondary sources. After the cache is created, subsequent builds pull only the change between commits. This mode is a good choice for projects with a clean working directory and a source that is a large Git repository. If you choose this option and your project does not use a Git repository (AWS CodeCommit, GitHub, GitHub Enterprise Server, or Bitbucket), the option is ignored. + Docker layer cache mode caches existing Docker layers. This mode is a good choice for projects that build or pull large Docker images. It can prevent the performance issues caused by pulling large Docker images down from the network. **Note** You can use a Docker layer cache in the Linux environment only. The `privileged` flag must be set so that your project has the required Docker permissions. By default, Docker daemon is enabled for non-VPC builds. If you would like to use Docker containers for VPC builds, see [Runtime Privilege and Linux Capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) on the Docker Docs website and enable privileged mode. Also, Windows does not support privileged mode. You should consider the security implication before you use a Docker layer cache. + Custom cache mode caches directories you specify in the buildspec file. This mode is a good choice if your build scenario is not suited to one of the other two local cache modes. If you use a custom cache: + Only directories can be specified for caching. You cannot specify individual files. + Symlinks are used to reference cached directories. + Cached directories are linked to your build before it downloads its project sources. Cached items overrides source items if they have the same name. Directories are specified using cache paths in the buildspec file. For more information, see [Buildspec syntax](build-spec-ref.md#build-spec-ref-syntax). + Avoid directory names that are the same in the source and in the cache. Locally-cached directories may override, or delete the contents of, directories in the source repository that have the same name. **Note** Local caching is not supported with the `LINUX_GPU_CONTAINER` environment type and the `BUILD_GENERAL1_2XLARGE` compute type. For more information, see [Build environment compute modes and types](build-env-ref-compute-types.md). **Note** Local caching is not supported when you configure CodeBuild to work with a VPC. For more information on using VPCs with CodeBuild, see [Use AWS CodeBuild with Amazon Virtual Private Cloud](vpc-support.md). # Specify a local cache You can use the AWS CLI, console, SDK, or CloudFormation to specify a local cache. For more information about local caching, see [Local caching](caching-local.md). **Topics** + [Specify local caching (CLI)](#caching-local-cli) + [Specify local caching (console)](#caching-local-console) + [Specify local caching (CloudFormation)](#caching-local-cfn) ## Specify local caching (CLI) You can use the the `--cache` parameter in the AWS CLI to specify each of the three local cache types. + To specify a source cache: ``` --cache type=LOCAL,mode=[LOCAL_SOURCE_CACHE] ``` + To specify a Docker layer cache: ``` --cache type=LOCAL,mode=[LOCAL_DOCKER_LAYER_CACHE] ``` + To specify a custom cache: ``` --cache type=LOCAL,mode=[LOCAL_CUSTOM_CACHE] ``` For more information, see [Create a build project (AWS CLI)](create-project.md#create-project-cli). ## Specify local caching (console) You specify a cache in the **Artifacts** section of the console. For **Cache type**, choose **Amazon S3** or **Local**. If you choose **Local**, choose one or more of the three local cache options. ![\[Specify a local cache by choosing one or more of the three local cache options.\]](http://docs.aws.amazon.com/codebuild/latest/userguide/images/local-cache.png) For more information, see [Create a build project (console)](create-project.md#create-project-console). ## Specify local caching (CloudFormation) If you use CloudFormation to specify a local cache, on the `Cache` property, for `Type`, specify `LOCAL`. The following sample YAML-formatted CloudFormation code specifies all three local cache types. You can specify any combination of the types. If you use a Docker layer cache, under `Environment`, you must set `PrivilegedMode` to `true` and `Type` to `LINUX_CONTAINER`. ``` CodeBuildProject: Type: AWS::CodeBuild::Project Properties: Name: MyProject ServiceRole: Artifacts: Type: S3 Location: Name: myArtifact EncryptionDisabled: true OverrideArtifactName: true Environment: Type: LINUX_CONTAINER ComputeType: BUILD_GENERAL1_SMALL Image: aws/codebuild/standard:5.0 Certificate: # PrivilegedMode must be true if you specify LOCAL_DOCKER_LAYER_CACHE PrivilegedMode: true Source: Type: GITHUB Location: InsecureSsl: true GitCloneDepth: 1 ReportBuildStatus: false TimeoutInMinutes: 10 Cache: Type: LOCAL Modes: # You can specify one or more cache mode, - LOCAL_CUSTOM_CACHE - LOCAL_DOCKER_LAYER_CACHE - LOCAL_SOURCE_CACHE ``` **Note** By default, Docker daemon is enabled for non-VPC builds. If you would like to use Docker containers for VPC builds, see [Runtime Privilege and Linux Capabilities](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) on the Docker Docs website and enable privileged mode. Also, Windows does not support privileged mode. For more information, see [Create a build project (CloudFormation)](create-project.md#create-project-cloud-formation).