Integrating CodeDeploy with Amazon EC2 Auto Scaling - AWS CodeDeploy
Deploying CodeDeploy applications to Amazon EC2 Auto Scaling groupsEnabling termination deployments during Auto Scaling scale-in eventsHow Amazon EC2 Auto Scaling works with CodeDeployUsing a custom AMI with CodeDeploy and Amazon EC2 Auto Scaling

Integrating CodeDeploy with Amazon EC2 Auto Scaling

CodeDeploy supports Amazon EC2 Auto Scaling, an AWS service that launches Amazon EC2 instances automatically according to conditions you define. These conditions can include limits exceeded in a specified time interval for CPU utilization, disk reads or writes, or inbound or outbound network traffic. Amazon EC2 Auto Scaling terminates the instances when they are no longer needed. For more information, see What is Amazon EC2 Auto Scaling? in the Amazon EC2 Auto Scaling User Guide.

When new Amazon EC2 instances are launched as part of an Amazon EC2 Auto Scaling group, CodeDeploy can deploy your revisions to the new instances automatically. You can also coordinate deployments in CodeDeploy with Amazon EC2 Auto Scaling instances registered with Elastic Load Balancing load balancers. For more information, see Integrating CodeDeploy with Elastic Load Balancing and Set up a load balancer in Elastic Load Balancing for CodeDeploy Amazon EC2 deployments.

Note

You might encounter issues if you associate multiple deployment groups with a single Amazon EC2 Auto Scaling group. If one deployment fails, for example, the instance will begin to shut down, but the other deployments that were running can take an hour to time out. For more information, see Avoid associating multiple deployment groups with a single Amazon EC2 Auto Scaling group and Under the hood: CodeDeploy and Amazon EC2 Auto Scaling integration.

Deploying CodeDeploy applications to Amazon EC2 Auto Scaling groups

To deploy a CodeDeploy application revision to an Amazon EC2 Auto Scaling group:

  1. Create or locate an IAM instance profile that allows the Amazon EC2 Auto Scaling group to work with Amazon S3. For more information, see Step 4: Create an IAM instance profile for your Amazon EC2 instances.

    Note

    You can also use CodeDeploy to deploy revisions from GitHub repositories to Amazon EC2 Auto Scaling groups. Although Amazon EC2 instances still require an IAM instance profile, the profile doesn't need any additional permissions to deploy from a GitHub repository.

  2. Create or use an Amazon EC2 Auto Scaling group, specifying the IAM instance profile in your launch configuration or template. For more information, see IAM role for applications that run on Amazon EC2 instances.

  3. Create or locate a service role that allows CodeDeploy to create a deployment group that contains the Amazon EC2 Auto Scaling group.

  4. Create a deployment group with CodeDeploy, specifying the Amazon EC2 Auto Scaling group name, the service role, and a few other options. For more information, see Create a deployment group for an in-place deployment (console) or Create a deployment group for an in-place deployment (console).

  5. Use CodeDeploy to deploy your revision to the deployment group that contains the Amazon EC2 Auto Scaling group.

For more information, see Tutorial: Use CodeDeploy to deploy an application to an Auto Scaling group.

Enabling termination deployments during Auto Scaling scale-in events

A termination deployment is a type of CodeDeploy deployment that is activated automatically when an Auto Scaling scale-in event occurs. CodeDeploy performs the termination deployment right before the Auto Scaling service terminates the instance. During a termination deployment, CodeDeploy doesn't deploy anything. Instead, it generates lifecycle events, which you can hook up to your own scripts to enable custom shutdown functionality. For example, you could hook up the ApplicationStop lifecycle event to a script that shuts down your application gracefully before the instance is terminated.

For a list of lifecycle events that CodeDeploy generates during a termination deployment, see Lifecycle event hook availability.

If the termination deployment fails for any reason, CodeDeploy will allow the instance termination to proceed. This means that the instance will be shut down even though CodeDeploy did not run the full set (or any) of the lifecycle events to completion.

If you don't enable termination deployments, the Auto Scaling service will still terminate Amazon EC2 instances when a scale-in event occurs, but CodeDeploy will not generate lifecycle events.

Note

Regardless of whether you enable termination deployments or not, if the Auto Scaling service terminates an Amazon EC2 instance while a CodeDeploy deployment is underway, then a race condition may occur between the lifecycle events generated by the Auto Scaling and CodeDeploy services. For example, the Terminating lifecycle event (generated by the Auto Scaling service) might override the ApplicationStart event (generated by the CodeDeploy deployment). In this scenario, you may experience a failure with either the Amazon EC2 instance termination or the CodeDeploy deployment.

To enable CodeDeploy to perform termination deployments

After the termination hook is installed, a scale-in (termination) event unfolds as follows:

  1. The Auto Scaling service (or simply, Auto Scaling) determines that a scale-in event needs to occur, and contacts the EC2 service to terminate an EC2 instance.

  2. The EC2 service starts terminating the EC2 instance. The instance moves into the Terminating state, and then into the Terminating:Wait state.

  3. During Terminating:Wait, Auto Scaling runs all the lifecycle hooks attached to the Auto Scaling group, including the termination hook installed by CodeDeploy.

  4. The termination hook sends a notification to the Amazon SQS queue that is polled by CodeDeploy.

  5. Upon receiving the notification, CodeDeploy parses the message, performs some validation, and performs a termination deployment.

  6. While the termination deployment is running, CodeDeploy sends heartbeats every five minutes to Auto Scaling to let it know that the instance is still being worked on.

  7. So far, the EC2 instance is still in the Terminating:Wait state (or possibly the Warmed:Pending:Wait state, if you've enabled Auto Scaling group warm pools).

  8. When the deployment completes, CodeDeploy indicates to Auto Scaling to CONTINUE the EC2 termination process, regardless of whether the termination deployment succeeded or failed.

How Amazon EC2 Auto Scaling works with CodeDeploy

When you create or update a CodeDeploy deployment group to include an Auto Scaling group, CodeDeploy accesses the Auto Scaling group using the CodeDeploy service role, and then installs Auto Scaling lifecycle hooks into your Auto Scaling groups.

Note

Auto Scaling lifecycle hooks are different from the lifecycle events (also called lifecycle event hooks) generated by CodeDeploy and described in the AppSpec 'hooks' section of this guide.

The Auto Scaling lifecycle hooks that CodeDeploy installs are:

  • A launch hook — This hook notifies CodeDeploy that an Auto Scaling scale-out event is in progress, and that CodeDeploy needs to start a launch deployment.

    During a launch deployment, CodeDeploy:

    • Deploys a revision of your application to the scaled-out instance.

    • Generates lifecycle events to indicate the progress of the deployment. You can hook up these lifecycle events to your own scripts to enable custom startup functionality. For more information, see the table in Lifecycle event hook availability.

    The launch hook and associated launch deployment are always enabled and cannot be turned off.

  • A termination hook — This optional hook notifies CodeDeploy that an Auto Scaling scale-in event is in progress, and that CodeDeploy needs to start a termination deployment.

    During a termination deployment, CodeDeploy generates lifecycle events to indicate the progress of the instance shutdown. For more information, see Enabling termination deployments during Auto Scaling scale-in events.

After CodeDeploy installs the lifecycle hooks, how are they used?

After the launch and termination lifecycle hooks are installed, they are used by CodeDeploy during Auto Scaling group scale-out and scale-in events, respectively.

A scale-out (launch) event unfolds as follows:

  1. The Auto Scaling service (or simply, Auto Scaling) determines that a scale-out event needs to occur, and contacts the EC2 service to launch a new EC2 instance.

  2. The EC2 service launches a new EC2 instance. The instance moves into the Pending state, and then into the Pending:Wait state.

  3. During Pending:Wait, Auto Scaling runs all the lifecycle hooks attached to the Auto Scaling group, including the launch hook installed by CodeDeploy.

  4. The launch hook sends a notification to the Amazon SQS queue that is polled by CodeDeploy.

  5. Upon receiving the notification, CodeDeploy parses the message, performs some validation, and starts a launch deployment.

  6. While the launch deployment is running, CodeDeploy sends heartbeats every five minutes to Auto Scaling to let it know that the instance is still being worked on.

  7. So far, the EC2 instance is still in the Pending:Wait state.

  8. When the deployment completes, CodeDeploy indicates to Auto Scaling to either CONTINUE or ABANDON the EC2 launch process, depending on whether the deployment succeeded or failed.

    • If CodeDeploy indicates CONTINUE, Auto Scaling continues the launch process, either waiting for other hooks to complete, or putting the instance into the Pending:Proceed and then the InService state.

    • If CodeDeploy indicates ABANDON, Auto Scaling terminates the EC2 instance, and restarts the launch procedure if needed to meet the desired number of instances, as defined in the Auto Scaling Desired Capacity setting.

A scale-in (termination) event unfolds as follows:

See Enabling termination deployments during Auto Scaling scale-in events.

How CodeDeploy names Amazon EC2 Auto Scaling groups

During blue/green deployments on an EC2/On-Premises compute platform, you have two options for adding instances to your replacement (green) environment:

  • Use instances that already exist or that you create manually.

  • Use settings from an Amazon EC2 Auto Scaling group that you specify to define and create instances in a new Amazon EC2 Auto Scaling group.

If you choose the second option, CodeDeploy provisions a new Amazon EC2 Auto Scaling group for you. It uses the following convention to name the group: 

CodeDeploy_deployment_group_name_deployment_id

For example, if a deployment with ID 10 deploys a deployment group named alpha-deployments, the provisioned Amazon EC2 Auto Scaling group is named CodeDeploy_alpha-deployments_10. For more information, see Create a deployment group for an EC2/On-Premises blue/green deployment (console) and GreenFleetProvisioningOption.

Execution order of custom lifecycle hook events

You can add your own lifecycle hooks to Amazon EC2 Auto Scaling groups to which CodeDeploy deploys. However, the order in which those custom lifecycle hook events are executed cannot be predetermined in relation to CodeDeploy default deployment lifecycle events. For example, if you add a custom lifecycle hook named ReadyForSoftwareInstall to an Amazon EC2 Auto Scaling group, you cannot know beforehand whether it will be executed before the first, or after the last, CodeDeploy default deployment lifecycle event.

To learn how to add custom lifecycle hooks to an Amazon EC2 Auto Scaling group, see Adding lifecycle hooks in the Amazon EC2 Auto Scaling User Guide.

Scale-out events during a deployment

If an Auto Scaling scale-out event occurs while a deployment is underway, the new instances will be updated with the application revision that was previously deployed, not the newest application revision. If the deployment succeeds, the old instances and the newly scaled-out instances will be hosting different application revisions. To bring the instances with the older revision up to date, CodeDeploy automatically starts a follow-on deployment (immediately after the first) to update any outdated instances. If you'd like to change this default behavior so that outdated EC2 instances are left at the older revision, see Automatic updates to outdated instances.

If you want to suspend Amazon EC2 Auto Scaling scale-out processes while deployments are taking place, you can do this through a setting in the common_functions.sh script that is used for load balancing with CodeDeploy. If HANDLE_PROCS=true, the following Auto Scaling events are suspended automatically during the deployment process:

  • AZRebalance

  • AlarmNotification

  • ScheduledActions

  • ReplaceUnhealthy

Important

Only the CodeDeployDefault.OneAtATime deployment configuration supports this functionality.

For more information about using HANDLE_PROCS=true to avoid deployment problems when using Amazon EC2 Auto Scaling, see Important notice about handling AutoScaling processes in aws-codedeploy-samples on GitHub.

Scale-in events during a deployment

If an Auto Scaling group starts scaling in while a CodeDeploy deployment is underway on that Auto Scaling group, a race condition could occur between the termination process (including the CodeDeploy termination deployment lifecycle events) and other CodeDeploy lifecycle events on the terminating instance. The deployment on that specific instance may fail if the instance is terminated before all CodeDeploy lifecycle events complete. Also, the overall CodeDeploy deployment may or may not fail, depending on how you've set your Minimum healthy hosts setting in your deployment configuration.

Order of events in AWS CloudFormation cfn-init scripts

If you use cfn-init (or cloud-init) to run scripts on newly provisioned Linux-based instances, your deployments might fail unless you strictly control the order of events that occur after the instance starts.

That order must be:

  1. The newly provisioned instance starts.

  2. All cfn-init bootstrapping scripts run to completion.

  3. The CodeDeploy agent starts.

  4. The latest application revision is deployed to the instance.

If the order of events is not carefully controlled, the CodeDeploy agent might start a deployment before all the scripts have finished running.

To control the order of events, use one of these best practices:

  • Install the CodeDeploy agent through a cfn-init script, placing it after all other scripts.

  • Include the CodeDeploy agent in a custom AMI and use a cfn-init script to start it, placing it after all other scripts.

For information about using cfn-init, see cfn-init in the AWS CloudFormation User Guide.

Using a custom AMI with CodeDeploy and Amazon EC2 Auto Scaling

You have two options for specifying the base AMI to use when new Amazon EC2 instances are launched in an Amazon EC2 Auto Scaling group:

  • You can specify a base custom AMI that already has the CodeDeploy agent installed. Because the agent is already installed, this option launches new Amazon EC2 instances more quickly than the other option. However, this option provides a greater likelihood that initial deployments of Amazon EC2 instances will fail, especially if the CodeDeploy agent is out of date. If you choose this option, we recommend you regularly update the CodeDeploy agent in your base custom AMI.

  • You can specify a base AMI that doesn't have the CodeDeploy agent installed and have the agent installed as each new instance is launched in an Amazon EC2 Auto Scaling group. Although this option launches new Amazon EC2 instances more slowly than the other option, it provides a greater likelihood that initial deployments of instances will succeed. This option uses the most recent version of the CodeDeploy agent.