AWS Data Pipeline examples using AWS CLI

The following code examples show you how to perform actions and implement common scenarios by using the AWS Command Line Interface with AWS Data Pipeline.

Actions are code excerpts from larger programs and must be run in context. While actions show you how to call individual service functions, you can see actions in context in their related scenarios.

Each example includes a link to the complete source code, where you can find instructions on how to set up and run the code in context.

Topics

Actions

Actions

The following code example shows how to use activate-pipeline.

AWS CLI

To activate a pipeline

The following activate-pipeline example activates the specified pipeline.


aws datapipeline activate-pipeline \
    --pipeline-id df-00627471SOVYZEXAMPLE

This command produces no output.

To activate the pipeline at a specific date and time, use the following command.


aws datapipeline activate-pipeline \
    --pipeline-id df-00627471SOVYZEXAMPLE \
    --start-timestamp 2015-04-07T00:00:00Z

For more information, see Working with pipelines in the AWS Data Pipeline Developer Guide.

For API details, see ActivatePipeline in AWS CLI Command Reference.

The following code example shows how to use add-tags.

AWS CLI

To add a tag to a pipeline

The following add-tags example adds the specified tag to the specified pipeline.


aws datapipeline add-tags \
    --pipeline-id df-00627471SOVYZEXAMPLE \
    --tags key=environment,value=production key=owner,value=sales

This command produces no output.

To view the tags, use the describe-pipelines command. For example, the tags added in the example command appear as follows in the output for describe-pipelines:


{
    ...
        "tags": [
            {
                "value": "production",
                "key": "environment"
            },
            {
                "value": "sales",
                "key": "owner"
            }
        ]
    ...
}

For more information, see Tagging Your Pipeline in the AWS Data Pipeline Developer Guide.

For API details, see AddTags in AWS CLI Command Reference.

The following code example shows how to use create-pipeline.

AWS CLI

To create a pipeline

The following create-pipeline example creates a pipeline.


aws datapipeline create-pipeline \
    --name my-pipeline \
    --unique-id my-pipeline-token

Output:


{
    "pipelineId": "df-00627471SOVYZEXAMPLE"
}

For more information, see Creating a pipeline in the AWS Data Pipeline Developer Guide.

For API details, see CreatePipeline in AWS CLI Command Reference.

The following code example shows how to use deactivate-pipeline.

AWS CLI

To deactivate a pipeline

The following deactivate-pipeline example deactivates the specified pipeline.


aws datapipeline deactivate-pipeline \
    --pipeline-id df-00627471SOVYZEXAMPLE

This command produces no output.

To deactivate the pipeline only after all running activities finish, use the following command.


aws datapipeline deactivate-pipeline \
    --pipeline-id df-00627471SOVYZEXAMPLE \
    --no-cancel-active

For more information, see Deactivating Your Pipeline in the AWS Data Pipeline Developer Guide.

For API details, see DeactivatePipeline in AWS CLI Command Reference.

The following code example shows how to use delete-pipeline.

AWS CLI

To delete a pipeline

The following delete-pipeline example deletes the specified pipeline.


aws datapipeline delete-pipeline \
    --pipeline-id df-00627471SOVYZEXAMPLE

This command produces no output.

For more information, see Deleting Your Pipeline in the AWS Data Pipeline Developer Guide.

For API details, see DeletePipeline in AWS CLI Command Reference.

The following code example shows how to use describe-pipelines.

AWS CLI

To describe your pipelines

The following describe-pipelines example describes the specified pipeline.


aws datapipeline describe-pipelines \
    --pipeline-ids df-00627471SOVYZEXAMPLE

Output:


{
    "pipelineDescriptionList": [
        {
            "fields": [
                {
                    "stringValue": "PENDING",
                    "key": "@pipelineState"
                },
                {
                    "stringValue": "my-pipeline",
                    "key": "name"
                },
                {
                    "stringValue": "2015-04-07T16:05:58",
                    "key": "@creationTime"
                },
                {
                    "stringValue": "df-00627471SOVYZEXAMPLE",
                    "key": "@id"
                },
                {
                    "stringValue": "123456789012",
                    "key": "pipelineCreator"
                },
                {
                    "stringValue": "PIPELINE",
                    "key": "@sphere"
                },
                {
                    "stringValue": "123456789012",
                    "key": "@userId"
                },
                {
                    "stringValue": "123456789012",
                    "key": "@accountId"
                },
                {
                    "stringValue": "my-pipeline-token",
                    "key": "uniqueId"
                }
            ],
            "pipelineId": "df-00627471SOVYZEXAMPLE",
            "name": "my-pipeline",
            "tags": []
        }
    ]
}

For more information, see Working with pipelines in the AWS Data Pipeline Developer Guide.

For API details, see DescribePipelines in AWS CLI Command Reference.

The following code example shows how to use get-pipeline-definition.

AWS CLI

To get a pipeline definition

The following get-pipeline-definition example gets the pipeline definition for the specified pipeline.


aws datapipeline get-pipeline-definition \
    --pipeline-id df-00627471SOVYZEXAMPLE

Output:


{
    "parameters": [
        {
            "type": "AWS::S3::ObjectKey",
            "id": "myS3OutputLoc",
            "description": "S3 output folder"
        },
        {
            "default": "s3://us-east-1.elasticmapreduce.samples/pig-apache-logs/data",
            "type": "AWS::S3::ObjectKey",
            "id": "myS3InputLoc",
            "description": "S3 input folder"
        },
        {
            "default": "grep -rc \"GET\" ${INPUT1_STAGING_DIR}/* > ${OUTPUT1_STAGING_DIR}/output.txt",
            "type": "String",
            "id": "myShellCmd",
            "description": "Shell command to run"
        }
    ],
    "objects": [
        {
            "type": "Ec2Resource",
            "terminateAfter": "20 Minutes",
            "instanceType": "t1.micro",
            "id": "EC2ResourceObj",
            "name": "EC2ResourceObj"
        },
        {
            "name": "Default",
            "failureAndRerunMode": "CASCADE",
            "resourceRole": "DataPipelineDefaultResourceRole",
            "schedule": {
                "ref": "DefaultSchedule"
            },
            "role": "DataPipelineDefaultRole",
            "scheduleType": "cron",
            "id": "Default"
        },
        {
            "directoryPath": "#{myS3OutputLoc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}",
            "type": "S3DataNode",
            "id": "S3OutputLocation",
            "name": "S3OutputLocation"
        },
        {
            "directoryPath": "#{myS3InputLoc}",
            "type": "S3DataNode",
            "id": "S3InputLocation",
            "name": "S3InputLocation"
        },
        {
            "startAt": "FIRST_ACTIVATION_DATE_TIME",
            "name": "Every 15 minutes",
            "period": "15 minutes",
            "occurrences": "4",
            "type": "Schedule",
            "id": "DefaultSchedule"
        },
        {
            "name": "ShellCommandActivityObj",
            "command": "#{myShellCmd}",
            "output": {
                "ref": "S3OutputLocation"
            },
            "input": {
                "ref": "S3InputLocation"
            },
            "stage": "true",
            "type": "ShellCommandActivity",
            "id": "ShellCommandActivityObj",
            "runsOn": {
                "ref": "EC2ResourceObj"
            }
        }
    ],
    "values": {
        "myS3OutputLoc": "s3://amzn-s3-demo-bucket/",
        "myS3InputLoc": "s3://us-east-1.elasticmapreduce.samples/pig-apache-logs/data",
        "myShellCmd": "grep -rc \"GET\" ${INPUT1_STAGING_DIR}/* > ${OUTPUT1_STAGING_DIR}/output.txt"
    }
}

For more information, see Pipeline definition file syntax in the AWS Data Pipeline Developer Guide.

For API details, see GetPipelineDefinition in AWS CLI Command Reference.

The following code example shows how to use list-pipelines.

AWS CLI

To list your pipelines

The following list-pipelines example lists your pipelines.


aws datapipeline list-pipelines

Output:


{
    "pipelineIdList": [
        {
            "id": "df-00627471SOVYZEXAMPLE",
            "name": "my-pipeline"
        },
        {
            "id": "df-09028963KNVMREXAMPLE",
            "name": "ImportDDB"
        },
        {
            "id": "df-0870198233ZYVEXAMPLE",
            "name": "CrossRegionDDB"
        },
        {
            "id": "df-00189603TB4MZEXAMPLE",
            "name": "CopyRedshift"
        }
    ]
}

For more information, see Viewing Your Pipelines in the AWS Data Pipeline Developer Guide.

For API details, see ListPipelines in AWS CLI Command Reference.

The following code example shows how to use list-runs.

AWS CLI

Example 1: To list your pipeline runs

The following list-runs example lists the runs for the specified pipeline.


aws datapipeline list-runs \
    --pipeline-id df-00627471SOVYZEXAMPLE

Output:


    Name                       Scheduled Start        Status                     ID                                              Started                Ended
    -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1.  EC2ResourceObj             2015-04-12T17:33:02    CREATING                   @EC2ResourceObj_2015-04-12T17:33:02             2015-04-12T17:33:10
2.  S3InputLocation            2015-04-12T17:33:02    FINISHED                   @S3InputLocation_2015-04-12T17:33:02            2015-04-12T17:33:09    2015-04-12T17:33:09
3.  S3OutputLocation           2015-04-12T17:33:02    WAITING_ON_DEPENDENCIES    @S3OutputLocation_2015-04-12T17:33:02           2015-04-12T17:33:09
4.  ShellCommandActivityObj    2015-04-12T17:33:02    WAITING_FOR_RUNNER         @ShellCommandActivityObj_2015-04-12T17:33:02    2015-04-12T17:33:09

For more information, see Interpreting Pipeline Status Details in the AWS Data Pipeline Developer Guide.

Example 2: To list the pipeline runs between the specified dates

The following list-runs example uses the --start-interval to specify the dates to include in the output.


aws datapipeline list-runs \
    --pipeline-id df-01434553B58A2SHZUKO5 \
    --start-interval 2017-10-07T00:00:00,2017-10-08T00:00:00

Output:


        Name                                                Scheduled Start      Status
        ID                                                  Started              Ended
----------------------------------------------------------------------------------------------------
    1.  EC2ResourceObj                                      2017-10-07T09:00:00  FINISHED
        @EC2ResourceObj_2017-10-07T09:00:00                 2017-10-07T09:00:05  2017-10-07T09:12:45

    2.  S3InputLocation                                     2017-10-07T09:00:00  FINISHED
        @S3InputLocation_2017-10-07T09:00:00                2017-10-07T09:01:10  2017-10-07T09:01:10

    3.  S3OutputLocation                                    2017-10-07T09:00:00  FINISHED
        @S3OutputLocation_2017-10-07T09:00:00               2017-10-07T09:01:10  2017-10-07T09:10:52

    4.  ShellCommandActivityObj                             2017-10-07T09:00:00  FINISHED
        @ShellCommandActivityObj_2017-10-07T09:00:00        2017-10-07T09:01:10  2017-10-07T09:10:52

For more information, see Viewing Pipeline Instance Details in the AWS Data Pipeline Developer Guide.

For API details, see ListRuns in AWS CLI Command Reference.

The following code example shows how to use put-pipeline-definition.

AWS CLI

To upload a pipeline definition

The following put-pipeline-definition example uploads the specified pipeline definition to the specified pipeline.


aws datapipeline put-pipeline-definition \
    --pipeline-id df-00627471SOVYZEXAMPLE \
    --pipeline-definition file://my-pipeline-definition.json

Contents of my-pipeline-definition.json:


{
    "parameters": [
        {
            "type": "AWS::S3::ObjectKey",
            "id": "myS3OutputLoc",
            "description": "S3 output folder"
        },
        {
            "default": "s3://amzn-s3-demo-bucket/pig-apache-logs/data",
            "type": "AWS::S3::ObjectKey",
            "id": "myS3InputLoc",
            "description": "S3 input folder"
        },
        {
            "default": "grep -rc \"GET\" ${INPUT1_STAGING_DIR}/* > ${OUTPUT1_STAGING_DIR}/output.txt",
            "type": "String",
            "id": "myShellCmd",
            "description": "Shell command to run"
        }
    ],
    "objects": [
        {
            "type": "Ec2Resource",
            "terminateAfter": "20 Minutes",
            "instanceType": "t1.micro",
            "id": "EC2ResourceObj",
            "name": "EC2ResourceObj"
        },
        {
            "name": "Default",
            "failureAndRerunMode": "CASCADE",
            "resourceRole": "DataPipelineDefaultResourceRole",
            "schedule": {
                "ref": "DefaultSchedule"
            },
            "role": "DataPipelineDefaultRole",
            "scheduleType": "cron",
            "id": "Default"
        },
        {
            "directoryPath": "#{myS3OutputLoc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}",
            "type": "S3DataNode",
            "id": "S3OutputLocation",
            "name": "S3OutputLocation"
        },
        {
            "directoryPath": "#{myS3InputLoc}",
            "type": "S3DataNode",
            "id": "S3InputLocation",
            "name": "S3InputLocation"
        },
        {
            "startAt": "FIRST_ACTIVATION_DATE_TIME",
            "name": "Every 15 minutes",
            "period": "15 minutes",
            "occurrences": "4",
            "type": "Schedule",
            "id": "DefaultSchedule"
        },
        {
            "name": "ShellCommandActivityObj",
            "command": "#{myShellCmd}",
            "output": {
                "ref": "S3OutputLocation"
            },
            "input": {
                "ref": "S3InputLocation"
            },
            "stage": "true",
            "type": "ShellCommandActivity",
            "id": "ShellCommandActivityObj",
            "runsOn": {
                "ref": "EC2ResourceObj"
            }
        }
    ],
    "values": {
        "myS3OutputLoc": "s3://amzn-s3-demo-bucket/",
        "myS3InputLoc": "s3://amzn-s3-demo-bucket/pig-apache-logs/data",
        "myShellCmd": "grep -rc \"GET\" ${INPUT1_STAGING_DIR}/* > ${OUTPUT1_STAGING_DIR}/output.txt"
    }
}

Output:


{
    "validationErrors": [],
    "errored": false,
    "validationWarnings": []
}

For more information, see Pipeline definition file syntax in the AWS Data Pipeline Developer Guide.

For API details, see PutPipelineDefinition in AWS CLI Command Reference.

The following code example shows how to use remove-tags.

AWS CLI

To remove a tag from a pipeline

The following remove-tags example removes the specified tag from the specified pipeline.


aws datapipeline remove-tags \
    --pipeline-id df-00627471SOVYZEXAMPLE \
    --tag-keys environment

This command produces no output.

For more information, see Tagging Your Pipeline in the AWS Data Pipeline Developer Guide.

For API details, see RemoveTags in AWS CLI Command Reference.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Amazon Data Lifecycle Manager

DataSync