Seller delivery data feeds service
AWS Marketplace provides data feeds as a mechanism to send structured, up-to-date product and customer information from AWS Marketplace systems to seller Amazon S3 buckets for ETL (extract, transform, and load) between seller-owned business intelligence tools.
The transactional data is delivered and appended in a bi-temporal structure so sellers can store and query data along two timelines with timestamps for both
-
valid time: when a fact occurred in the real world (“what you knew”)
-
system time: when that fact was recorded to the database (“when you knew it”).
Data feeds are delivered daily at midnight UTC following an update from the prior day containing 24 hours of data from the previous day. An update can be defined by a customer subscribing, a customer being invoiced, or AWS disbursing payment.
This section provides an overview of data feeds and explains how to access and use them. Subsequent sections describe each data feed.
Storage and structure of data feeds
Data feeds collect and deliver comma-separated value (CSV) files to an encrypted Amazon S3 bucket that you provide. The CSV files have the following characteristics:
-
They follow 4180 standards
. -
Character encoding is UTF-8 without BOM.
-
Commas are used as separators between values.
-
Fields are escaped by double quotation marks.
-
\n
is the line feed character. -
Dates are reported in the UTC time zone, are in ISO 8601 date and time format, and are accurate within 1 second.
-
All
*_period_start_date
and*_period_end_date
values are inclusive, which means that23:59:59
is the last possible timestamp for any day. -
All monetary fields are preceded with a currency field.
-
Monetary fields use a period (
.
) character as a decimal separator, and don't use a comma (,) as a thousands separator.
Data feeds are generated and stored as follows:
-
Data feeds are generated within a day, and contain 24 hours of data from the previous day.
-
In the Amazon S3 bucket, data feeds are organized by month using the following format:
bucket-name
/data-feed-name_version
/year=YYYY
/month=MM
/data.csv -
As each daily data feed is generated, it is appended to the existing CSV file for that month. When a new month starts, a new CSV file is generated for each data feed.
-
Information in data feeds is backfilled from 2010/01/01 to 2020/04/30 (inclusive) and is available in the CSV file in the
year=2010/month=01
subfolder.You may notice cases where the current month's file for a given data feed contains only column headers, and no data. This means that there were no new entries for that month for the feed. This can happen with data feeds that are updated less frequently, like the product feed. In these cases, data is available in the backfilled folder.
-
In Amazon S3, you can create an Amazon S3 lifecycle policy to manage how long to keep files in the bucket.
-
You can configure Amazon SNS to notify you when data is delivered to your encrypted S3 bucket. For information on how to configure notifications, see Getting started with Amazon SNS in the Amazon Simple Notification Service Developer Guide.
Historization of the data
Each data feed includes columns that document the history of the data. Except for
valid_to
, these columns are common to all data feeds. They're included as a
common history schema and are useful in querying the data.
Column name | Description |
---|---|
valid_from | The first date that the value for the primary key is valid for in relation to values for other fields. |
valid_to | This column is only shown on the Address data feed and is always blank. |
insert_date | The date a record was inserted into the data feed. |
update_date | The date the record was last updated. |
delete_date | This column is always blank. |
The following shows an example of these columns.
valid_from | valid_to | insert_date | update_date | delete_date |
---|---|---|---|---|
2018-12-12T02:00:00Z | 2018-12-12T02:00:00Z | 2018-12-12T02:00:00Z | ||
2019-03-29T03:00:00Z | 2019-03-29T03:00:00Z | 2019-03-29T03:00:00Z | ||
2019-03-29T03:00:00Z | 2019-03-29T03:00:00Z | 2019-04-28T03:00:00Z |
The valid_from
and update_date
field together form a
bi-temporal data model. The valid_from
field, as it is
named, tells you when the item is valid from. If the item was edited, it can have multiple
records in the feed, each with a different update_date
, but the same
valid_from
date. For example, to find the current value for an item, you
would find the record with the most recent update_date
, from the list of
records with the most recent valid_from
date.
In the example above, the record was originally created 2018-12-12. It was then changed
on 2019-03-29 (for example, if the address in the record changed). Later, on 2019-04-28, the
address change was corrected (so the valid_from
didn't change, but the
update_date
did). Correcting the address (a rare event) retroactively changes
the record from the original valid_from
date, so that field didn't change. A
query to find the most recent valid_from
would return two records, the one with
the latest update_date
gives you the actual current record.
Accessing data feeds
To access data feeds, you need to configure your environment to receive data feeds to an encrypted Amazon S3 bucket. You follow a multi-step process to access data feeds, and the following sets of steps explain how.
Allocate a business intelligence or data engineer with SQL and ETL (extract, transform, load) experience. This person also needs experience setting up APIs.
Set up an Amazon Simple Storage Service bucket and a subscription to the data feeds. Use the AWS seller account ID associated with your Marketplace product listings. To do so, you can watch this YouTube video
or follow the steps below. The video and the steps explain how to use a AWS CloudFormation template
that helps simplify configuration. -
Open a web browser and sign into the AWS Marketplace Management Portal
, then go to Set up customer data storage . -
Choose Create resources with AWS CloudFormation template to open the template in the AWS CloudFormation console in another window.
-
In the template, specify the following and then choose Next:
-
Stack name – The collection of resources you're creating to enable access to data feeds.
-
Amazon S3 bucket name – The bucket for storing data feeds.
-
(Optional) Amazon SNS topic name – The topic for receiving notifications when AWS delivers new data to the Amazon S3 bucket.
-
-
On the Review page, confirm your entries and choose Create stack. This will open a new page with the CloudFormation status and details.
-
From the Resources tab, copy Amazon Resource Names (ARNs) for the following resources from the CloudFormation page into the fields on the AWS Marketplace Set up customer data storage
page: -
Amazon S3 bucket for storing data feeds
-
AWS KMS key for encrypting the Amazon S3 bucket
-
(Optional) Amazon SNS topic for receiving notifications when AWS delivers new data to the Amazon S3 bucket
-
-
On the Set up customer data storage page, choose Submit.
-
(Optional) Edit the policies created by the CloudFormation template. See Data feed policies for more details.
You are now subscribed to data feeds. The next time data feeds are generated, you can access the data.
-
Use an ETL (extract, transform, load) operation to connect the data feeds to your data warehouse or relational database.
Note
Data tools have different capabilities. You must involve a business intelligence engineer or data engineer to set up the integration to match your tool’s capabilities.
To run or create SQL queries, configure the data feeds to enforce primary and foreign keys in your data tool. Each data feed represents a unique table, and you must set up all data feeds in data schema with the entity relationships. For more information about the tables and entity relationships, see Data feed tables overview in this guide.
Setup Amazon Simple Notification Service to automatically refresh your data warehouse or relational database. You can configure Amazon SNS notifications to send alerts when data from each unique feed is delivered to an S3 bucket. These notifications can be leveraged to automatically refresh seller data warehouse when new data is received via data feeds, if seller data tool supports this capability. See Getting started with Amazon SNS in the Amazon Simple Notification Service Developer Guide.
Example notification:
{ "mainExecutionId": "
1bc08b11-ab4b-47e1-866a-9c8f38423a98
", "executionId": "52e862a9-42d2-41e0-8010-810af84d39b1
", "subscriptionId": "27ae3961-b13a-44bc-a1a7-365b2dc181fd
", "processedFiles": [], "executionStatus": "SKIPPED
", "errors": [], "feedType": "[data feed name
]" }Notifications can have the following
executionStatus
states:SKIPPED
– The seller has no new data for the day.COMPLETED
– We delivered the feed with new data.FAILED
– The feed delivery has an issue.
Validate the setup by running SQL queries. You can use the sample queries in this guide, or the queries on GitHub, at https://github.com/aws-samples/aws-marketplace-api-samples/tree/main/seller-data-feeds/queries
. Note
The sample queries in this guide were written for AWS Athena. You may need to modify the queries for use with your tools.
-
Determine where business users want to consume data. For example, you can:
Export .csv data from your data warehouse or SQL database.
Connect your data to a visualization tool such as PowerBI or Tableau.
Map data to your CRM, ERP, or financial tools, such as Salesforce, Infor, or Netsuite.
For more information about AWS CloudFormation templates, see Working with AWS CloudFormation templates in the AWS CloudFormation User Guide.
Data feed policies
When your Amazon S3 bucket is created by the CloudFormation template, it will create policies for access attached to that bucket, the AWS KMS key, and the Amazon SNS topic. The policies allow the AWS Marketplace reports service to write to your bucket and SNS topic with the data feed information. Each policy will have a section like the following (this example is from the Amazon S3 bucket).
{ "Sid": "AwsMarketplaceDataFeedsAccess", "Effect": "Allow", "Principal": { "Service": "reports.marketplace.amazonaws.com" }, "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:GetEncryptionConfiguration", "s3:GetBucketAcl", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::
datafeed-bucket
", "arn:aws:s3:::datafeed-bucket
/*" ] },
In this policy, reports.marketplace.amazonaws.com
is the service principal
that AWS Marketplace uses to push data to the Amazon S3 bucket. The datafeed-bucket
is the bucket that you specified in the CloudFormation template.
When the AWS Marketplace reports service calls Amazon S3, AWS KMS, or Amazon SNS, it will provide the ARN of
the data it is intending to write to the bucket when it does. To ensure that the only data
written to your bucket is data written on your behalf, you can specify the
aws:SourceArn
in the condition of the policy. In the following example, you
must replace the account-id
with the ID for your
AWS account.
{ "Sid": "AwsMarketplaceDataFeedsAccess", "Effect": "Allow", "Principal": { "Service": "reports.marketplace.amazonaws.com" }, "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:GetEncryptionConfiguration", "s3:GetBucketAcl", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::datafeed-amzn-s3-demo-bucket", "arn:aws:s3:::datafeed-amzn-s3-demo-bucket/*" , "Condition": { "StringEquals": { "aws:SourceAccount": "
account-id
", "aws:SourceArn": ["arn:aws:marketplace::account-id
:AWSMarketplace/SellerDataSubscription/DataFeeds_V1", "arn:aws:marketplace::account-id
:AWSMarketplace/SellerDataSubscription/Example-Report
"] } } },
Unsubscribing from data feeds
Open a web browser and sign in to the AWS Marketplace Management
Portal
Using data feeds
When data is available in your Amazon S3 bucket, you can use data feeds in the following ways:
-
Download the .CSV files from the Amazon S3 bucket you created in Accessing data feeds so that you can view the data in a spreadsheet.
-
Use ETL (extract, transform, and load), SQL query, business analytics tools to collect and analyze the data.
You can use AWS services to collect and analyze data, or any third-party tool that can perform analysis of .CSV-based datasets.
Example: Use AWS services to collect and analyze data
The following procedure assumes that you've already configured your environment to receive data feeds to an Amazon S3 bucket and that the bucket contains data feeds.
To collect and analyze data from data feeds
-
From the AWS Glue console
, create a crawler to connect to the Amazon S3 bucket that stores the data feeds, extract the data you want, and create metadata tables in the AWS Glue Data Catalog. For more information about AWS Glue, see the AWS Glue Developer Guide.
-
From the Athena console
, run SQL queries on the data in the AWS Glue Data Catalog. For more information about Athena see the Amazon Athena User Guide.
-
From the Amazon QuickSight console
, create an analysis and then create a visual of the data. For more information about Amazon QuickSight, see the Amazon QuickSight User Guide.
For a detailed example of one way to use AWS services to collect and analyze data in
data feeds, see Using Seller Data Feed Delivery Service, Amazon Athena, and Amazon QuickSight to create seller
reports