Setting up for AWS Data Pipeline - AWS Data Pipeline
Sign up for AWSCreate IAM Roles for AWS Data Pipeline and Pipeline ResourcesAllow IAM Principals (Users and Groups) to Perform Necessary ActionsGranting programmatic access

AWS Data Pipeline is no longer available to new customers. Existing customers of AWS Data Pipeline can continue to use the service as normal. Learn more

Setting up for AWS Data Pipeline

Before you use AWS Data Pipeline for the first time, complete the following tasks.

After you complete these tasks, you can start using AWS Data Pipeline. For a basic tutorial, see Getting Started with AWS Data Pipeline.

Sign up for AWS

When you sign up for Amazon Web Services (AWS), your AWS account is automatically signed up for all services in AWS, including AWS Data Pipeline. You are charged only for the services that you use. For more information about AWS Data Pipeline usage rates, see AWS Data Pipeline.

Sign up for an AWS account

If you do not have an AWS account, complete the following steps to create one.

To sign up for an AWS account

  1. Open https://portal.aws.amazon.com/billing/signup.

  2. Follow the online instructions.

    Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.

    When you sign up for an AWS account, an AWS account root user is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform tasks that require root user access.

AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to https://aws.amazon.com/ and choosing My Account.

Create a user with administrative access

After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.

Secure your AWS account root user

  1. Sign in to the AWS Management Console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.

    For help signing in by using root user, see Signing in as the root user in the AWS Sign-In User Guide.

  2. Turn on multi-factor authentication (MFA) for your root user.

    For instructions, see Enable a virtual MFA device for your AWS account root user (console) in the IAM User Guide.

Create a user with administrative access

  1. Enable IAM Identity Center.

    For instructions, see Enabling AWS IAM Identity Center in the AWS IAM Identity Center User Guide.

  2. In IAM Identity Center, grant administrative access to a user.

    For a tutorial about using the IAM Identity Center directory as your identity source, see Configure user access with the default IAM Identity Center directory in the AWS IAM Identity Center User Guide.

Sign in as the user with administrative access

  • To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.

    For help signing in using an IAM Identity Center user, see Signing in to the AWS access portal in the AWS Sign-In User Guide.

Assign access to additional users

  1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.

    For instructions, see Create a permission set in the AWS IAM Identity Center User Guide.

  2. Assign users to a group, and then assign single sign-on access to the group.

    For instructions, see Add groups in the AWS IAM Identity Center User Guide.

Create IAM Roles for AWS Data Pipeline and Pipeline Resources

AWS Data Pipeline requires IAM roles that determine the permissions to perform actions and access AWS resources. The pipeline role determines the permissions that AWS Data Pipeline has, and a resource role determines the permissions that applications running on pipeline resources, such as EC2 instances, have. You specify these roles when you create a pipeline. Even if you do not specify a custom role and use the default roles DataPipelineDefaultRole and DataPipelineDefaultResourceRole, you must first create the roles and attach permissions policies. For more information, see IAM Roles for AWS Data Pipeline.

Allow IAM Principals (Users and Groups) to Perform Necessary Actions

To work with a pipeline, an IAM principal (a user or group) in your account must be allowed to perform required AWS Data Pipeline actions and actions for other services as defined by your pipeline.

To simplify permissions, the AWSDataPipeline_FullAccess managed policy is available for you to attach to IAM principals. This managed policy allows the principal to perform all actions that a user requires and the iam:PassRole action on the default roles used with AWS Data Pipeline when a custom role is not specified.

We highly recommend that you carefully evaluate this managed policy and restrict permissions only to those that your users require. If necessary, use this policy as a starting point, and then remove permissions to create a more restrictive inline permissions policy that you can attach to IAM principals. For more information and example permissions policies, see Example Policies for AWS Data Pipeline

A policy statement similar to the following example must be included in a policy attached to any IAM principal that uses the pipeline. This statement allows the IAM principal to perform the PassRole action on the roles that a pipeline uses. If you do not use default roles, replace MyPipelineRole and MyResourceRole with the custom roles that you create.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "iam:PassRole",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:iam::*:role/MyPipelineRole",
                "arn:aws:iam::*:role/MyResourceRole"
            ]
        }
    ]
}

The following procedure demonstrates how to create an IAM group, attach the AWSDataPipeline_FullAccess managed policy to the group, and then add users to the group. You can use this procedure for any inline policy

To create a user group DataPipelineDevelopers and attach the AWSDataPipeline_FullAccess policy

  1. Open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Groups, Create New Group.

  3. Enter a Group Name, for example, DataPipelineDevelopers, and then choose Next Step.

  4. Enter AWSDataPipeline_FullAccess for Filter and then select it from the list.

  5. Choose Next Step and then choose Create Group.

  6. To add users to the group:

    1. Select the group you created from the list of groups.

    2. Choose Group Actions, Add Users to Group.

    3. Select the users you want to add from the list and then choose Add Users to Group.

Granting programmatic access

Users need programmatic access if they want to interact with AWS outside of the AWS Management Console. The way to grant programmatic access depends on the type of user that's accessing AWS.

To grant users programmatic access, choose one of the following options.

Which user needs programmatic access? To By

Workforce identity

(Users managed in IAM Identity Center)

 Use temporary credentials to sign programmatic requests to the AWS CLI, AWS SDKs, or AWS APIs.

Following the instructions for the interface that you want to use.
IAM Use temporary credentials to sign programmatic requests to the AWS CLI, AWS SDKs, or AWS APIs. Following the instructions in Using temporary credentials with AWS resources in the IAM User Guide.
IAM

(Not recommended)

Use long-term credentials to sign programmatic requests to the AWS CLI, AWS SDKs, or AWS APIs.

Following the instructions for the interface that you want to use.