AWS Data Pipeline is no longer available to new customers. Existing customers of AWS Data Pipeline can continue to use the service as normal. [Learn more](https://aws.amazon.com/blogs/big-data/migrate-workloads-from-aws-data-pipeline/)
# Data Formats
The following are the AWS Data Pipeline data format objects:
**Topics**
+ [CSV Data Format](dp-object-csv.md)
+ [Custom Data Format](dp-object-custom.md)
+ [DynamoDBDataFormat](dp-object-dynamodbdataformat.md)
+ [DynamoDBExportDataFormat](dp-object-dynamodbexportdataformat.md)
+ [RegEx Data Format](dp-object-regex.md)
+ [TSV Data Format](dp-object-tsv.md)
# CSV Data Format
A comma-delimited data format where the column separator is a comma and the record separator is a newline character.
## Example
The following is an example of this object type.
```
{
"id" : "MyOutputDataType",
"type" : "CSV",
"column" : [
"Name STRING",
"Score INT",
"DateOfBirth TIMESTAMP"
]
}
```
## Syntax
****
| Optional Fields | Description | Slot Type |
| --- | --- | --- |
| column | Column name with datatype specified by each field for the data described by this data node. Ex: hostname STRING For multiple values, use column names and data types separated by a space. | String |
| escapeChar | A character, for example "\$1", that instructs the parser to ignore the next character. | String |
| parent | Parent of the current object from which slots will be inherited. | Reference Object, e.g. "parent":\$1"ref":"myBaseObjectId"\$1 |
****
| Runtime Fields | Description | Slot Type |
| --- | --- | --- |
| @version | Pipeline version the object was created with. | String |
****
| System Fields | Description | Slot Type |
| --- | --- | --- |
| @error | Error describing the ill-formed object | String |
| @pipelineId | Id of the pipeline to which this object belongs to | String |
| @sphere | The sphere of an object denotes its place in the lifecycle: Component Objects give rise to Instance Objects which execute Attempt Objects | String |
# Custom Data Format
A custom data format defined by a combination of a certain column separator, record separator, and escape character.
## Example
The following is an example of this object type.
```
{
"id" : "MyOutputDataType",
"type" : "Custom",
"columnSeparator" : ",",
"recordSeparator" : "\n",
"column" : [
"Name STRING",
"Score INT",
"DateOfBirth TIMESTAMP"
]
}
```
## Syntax
****
| Required Fields | Description | Slot Type |
| --- | --- | --- |
| columnSeparator | A character that indicates the end of a column in a data file. | String |
****
| Optional Fields | Description | Slot Type |
| --- | --- | --- |
| column | Column name with datatype specified by each field for the data described by this data node. Ex: hostname STRING For multiple values, use column names and data types separated by a space. | String |
| parent | Parent of the current object from which slots will be inherited. | Reference Object, e.g. "parent":\$1"ref":"myBaseObjectId"\$1 |
| recordSeparator | A character that indicates the end of a row in a data file, for example "\$1n". Only single characters are supported. | String |
****
| Runtime Fields | Description | Slot Type |
| --- | --- | --- |
| @version | Pipeline version the object was created with. | String |
****
| System Fields | Description | Slot Type |
| --- | --- | --- |
| @error | Error describing the ill-formed object | String |
| @pipelineId | Id of the pipeline to which this object belongs to | String |
| @sphere | The sphere of an object denotes its place in the lifecycle: Component Objects give rise to Instance Objects which execute Attempt Objects | String |
# DynamoDBDataFormat
Applies a schema to a DynamoDB table to make it accessible by a Hive query. `DynamoDBDataFormat` is used with a `HiveActivity` object and a `DynamoDBDataNode` input and output. `DynamoDBDataFormat` requires that you specify all columns in your Hive query. For more flexibility to specify certain columns in a Hive query or Amazon S3 support, see [DynamoDBExportDataFormat](dp-object-dynamodbexportdataformat.md).
**Note**
DynamoDB Boolean types are not mapped to Hive Boolean types. However, it is possible to map DynamoDB integer values of 0 or 1 to Hive Boolean types.
## Example
The following example shows how to use `DynamoDBDataFormat` to assign a schema to a `DynamoDBDataNode` input, which allows a `HiveActivity` object to access the data by named columns and copy the data to a `DynamoDBDataNode` output.
```
{
"objects": [
{
"id" : "Exists.1",
"name" : "Exists.1",
"type" : "Exists"
},
{
"id" : "DataFormat.1",
"name" : "DataFormat.1",
"type" : "DynamoDBDataFormat",
"column" : [
"hash STRING",
"range STRING"
]
},
{
"id" : "DynamoDBDataNode.1",
"name" : "DynamoDBDataNode.1",
"type" : "DynamoDBDataNode",
"tableName" : "$INPUT_TABLE_NAME",
"schedule" : { "ref" : "ResourcePeriod" },
"dataFormat" : { "ref" : "DataFormat.1" }
},
{
"id" : "DynamoDBDataNode.2",
"name" : "DynamoDBDataNode.2",
"type" : "DynamoDBDataNode",
"tableName" : "$OUTPUT_TABLE_NAME",
"schedule" : { "ref" : "ResourcePeriod" },
"dataFormat" : { "ref" : "DataFormat.1" }
},
{
"id" : "EmrCluster.1",
"name" : "EmrCluster.1",
"type" : "EmrCluster",
"schedule" : { "ref" : "ResourcePeriod" },
"masterInstanceType" : "m1.small",
"keyPair" : "$KEYPAIR"
},
{
"id" : "HiveActivity.1",
"name" : "HiveActivity.1",
"type" : "HiveActivity",
"input" : { "ref" : "DynamoDBDataNode.1" },
"output" : { "ref" : "DynamoDBDataNode.2" },
"schedule" : { "ref" : "ResourcePeriod" },
"runsOn" : { "ref" : "EmrCluster.1" },
"hiveScript" : "insert overwrite table ${output1} select * from ${input1} ;"
},
{
"id" : "ResourcePeriod",
"name" : "ResourcePeriod",
"type" : "Schedule",
"period" : "1 day",
"startDateTime" : "2012-05-04T00:00:00",
"endDateTime" : "2012-05-05T00:00:00"
}
]
}
```
## Syntax
****
| Optional Fields | Description | Slot Type |
| --- | --- | --- |
| column | The column name with data type specified by each field for the data described by this data node. For example, hostname STRING. For multiple values, use column names and data types separated by a space. | String |
| parent | The parent of the current object from which slots will be inherited. | Reference Object, such as "parent":\$1"ref":"myBaseObjectId"\$1 |
****
| Runtime Fields | Description | Slot Type |
| --- | --- | --- |
| @version | The pipeline version uses to create the object. | String |
****
| System Fields | Description | Slot Type |
| --- | --- | --- |
| @error | The error describing the ill-formed object. | String |
| @pipelineId | The Id of the pipeline to which this object belongs. | String |
| @sphere | The sphere of an object denotes its place in the lifecycle: Component Objects give rise to Instance Objects which execute Attempt Objects. | String |
# DynamoDBExportDataFormat
Applies a schema to an DynamoDB table to make it accessible by a Hive query. Use `DynamoDBExportDataFormat` with a `HiveCopyActivity` object and `DynamoDBDataNode` or `S3DataNode` input and output. `DynamoDBExportDataFormat` has the following benefits:
+ Provides both DynamoDB and Amazon S3 support
+ Allows you to filter data by certain columns in your Hive query
+ Exports all attributes from DynamoDB even if you have a sparse schema
**Note**
DynamoDB Boolean types are not mapped to Hive Boolean types. However, it is possible to map DynamoDB integer values of 0 or 1 to Hive Boolean types.
## Example
The following example shows how to use `HiveCopyActivity` and `DynamoDBExportDataFormat` to copy data from one `DynamoDBDataNode` to another, while filtering based on a time stamp.
```
{
"objects": [
{
"id" : "DataFormat.1",
"name" : "DataFormat.1",
"type" : "DynamoDBExportDataFormat",
"column" : "timeStamp BIGINT"
},
{
"id" : "DataFormat.2",
"name" : "DataFormat.2",
"type" : "DynamoDBExportDataFormat"
},
{
"id" : "DynamoDBDataNode.1",
"name" : "DynamoDBDataNode.1",
"type" : "DynamoDBDataNode",
"tableName" : "item_mapped_table_restore_temp",
"schedule" : { "ref" : "ResourcePeriod" },
"dataFormat" : { "ref" : "DataFormat.1" }
},
{
"id" : "DynamoDBDataNode.2",
"name" : "DynamoDBDataNode.2",
"type" : "DynamoDBDataNode",
"tableName" : "restore_table",
"region" : "us_west_1",
"schedule" : { "ref" : "ResourcePeriod" },
"dataFormat" : { "ref" : "DataFormat.2" }
},
{
"id" : "EmrCluster.1",
"name" : "EmrCluster.1",
"type" : "EmrCluster",
"schedule" : { "ref" : "ResourcePeriod" },
"masterInstanceType" : "m1.xlarge",
"coreInstanceCount" : "4"
},
{
"id" : "HiveTransform.1",
"name" : "Hive Copy Transform.1",
"type" : "HiveCopyActivity",
"input" : { "ref" : "DynamoDBDataNode.1" },
"output" : { "ref" : "DynamoDBDataNode.2" },
"schedule" : { "ref" : "ResourcePeriod" },
"runsOn" : { "ref" : "EmrCluster.1" },
"filterSql" : "`timeStamp` > unix_timestamp(\"#{@scheduledStartTime}\", \"yyyy-MM-dd'T'HH:mm:ss\")"
},
{
"id" : "ResourcePeriod",
"name" : "ResourcePeriod",
"type" : "Schedule",
"period" : "1 Hour",
"startDateTime" : "2013-06-04T00:00:00",
"endDateTime" : "2013-06-04T01:00:00"
}
]
}
```
## Syntax
****
| Optional Fields | Description | Slot Type |
| --- | --- | --- |
| column | Column name with datatype specified by each field for the data described by this data node. Ex: hostname STRING | String |
| parent | Parent of the current object from which slots will be inherited. | Reference Object, e.g. "parent":\$1"ref":"myBaseObjectId"\$1 |
****
| Runtime Fields | Description | Slot Type |
| --- | --- | --- |
| @version | Pipeline version the object was created with. | String |
****
| System Fields | Description | Slot Type |
| --- | --- | --- |
| @error | Error describing the ill-formed object | String |
| @pipelineId | Id of the pipeline to which this object belongs to | String |
| @sphere | The sphere of an object denotes its place in the lifecycle: Component Objects give rise to Instance Objects which execute Attempt Objects | String |
# RegEx Data Format
A custom data format defined by a regular expression.
## Example
The following is an example of this object type.
```
{
"id" : "MyInputDataType",
"type" : "RegEx",
"inputRegEx" : "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\"))?",
"outputFormat" : "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s",
"column" : [
"host STRING",
"identity STRING",
"user STRING",
"time STRING",
"request STRING",
"status STRING",
"size STRING",
"referer STRING",
"agent STRING"
]
}
```
## Syntax
****
| Optional Fields | Description | Slot Type |
| --- | --- | --- |
| column | Column name with datatype specified by each field for the data described by this data node. Ex: hostname STRING For multiple values, use column names and data types separated by a space. | String |
| inputRegEx | The regular expression to parse an S3 input file. inputRegEx provides a way to retrieve columns from relatively unstructured data in a file. | String |
| outputFormat | The column fields retrieved by inputRegEx, but referenced as %1\$1s %2\$1s using Java formatter syntax. | String |
| parent | Parent of the current object from which slots will be inherited. | Reference Object, e.g. "parent":\$1"ref":"myBaseObjectId"\$1 |
****
| Runtime Fields | Description | Slot Type |
| --- | --- | --- |
| @version | Pipeline version the object was created with. | String |
****
| System Fields | Description | Slot Type |
| --- | --- | --- |
| @error | Error describing the ill-formed object | String |
| @pipelineId | Id of the pipeline to which this object belongs to | String |
| @sphere | The sphere of an object denotes its place in the lifecycle: Component Objects give rise to Instance Objects which execute Attempt Objects | String |
# TSV Data Format
A comma-delimited data format where the column separator is a tab character and the record separator is a newline character.
## Example
The following is an example of this object type.
```
{
"id" : "MyOutputDataType",
"type" : "TSV",
"column" : [
"Name STRING",
"Score INT",
"DateOfBirth TIMESTAMP"
]
}
```
## Syntax
****
| Optional Fields | Description | Slot Type |
| --- | --- | --- |
| column | Column name and data type for the data described by this data node. For example "Name STRING" denotes a column named Name with fields of data type STRING. Separate multiple column name and data type pairs with commas (as shown in the example). | String |
| columnSeparator | The character that separates fields in one column from fields in the next column. Defaults to '\$1t'. | String |
| escapeChar | A character, for example "\$1", that instructs the parser to ignore the next character. | String |
| parent | Parent of the current object from which slots are inherited. | Reference Object, for example, "parent":\$1"ref":"myBaseObjectId"\$1 |
| recordSeparator | The character that separates records. Defaults to '\$1n'. | String |
****
| Runtime Fields | Description | Slot Type |
| --- | --- | --- |
| @version | Pipeline version that the object was created with. | String |
****
| System Fields | Description | Slot Type |
| --- | --- | --- |
| @error | Error describing the ill-formed object. | String |
| @pipelineId | ID of the pipeline to which this object belongs. | String |
| @sphere | The sphere of an object denotes its place in the lifecycle: Component Objects give rise to Instance Objects, which execute Attempt Objects. | String |