Amazon Monitron is no longer open to new customers. Existing customers can continue to use the service as normal. For capabilities similar to Amazon Monitron, see our [blog post](https://aws.amazon.com/blogs/machine-learning/maintain-access-and-consider-alternatives-for-amazon-monitron). # Accessing your Amazon Monitron data There are two ways to access your raw Amazon Monitron data outside of Amazon Monitron. You may want access your data on an ongoing basis, so that you can use it elsewhere. In that case, you can configure Amazon Monitron to automatically [add your data to a Kinesis stream](/Monitron/latest/user-guide/monitron-kinesis-export.html). From there, you can port it to various destinations, including Amazon S3 and Lambda. This process requires configuration, and that configuration requires an understanding of Kinesis Data Streams. However, once you have all the elements arranged to your satisfaction, you can keep your data streaming automatically. Or you may want to access your data once in a while, just to gain a clear understanding of what kind of data you are storing and analyzing on AWS. In that case, you can ask AWS support to [manually copy your data to Amazon S3](data-download-monitron.md). This process requires less configuration, but it cannot be automated. It only gives you the data that Amazon Monitron has accumulated up until now, in one chunk. **Topics** + [Exporting your Amazon Monitron data to Amazon S3](data-download-monitron.md) + [Amazon Monitron Kinesis data export v1](monitron-kinesis-export.md) + [Amazon Monitron Kinesis data export v2](monitron-kinesis-export-v2.md) # Exporting your Amazon Monitron data to Amazon S3 You may sometimes want to access the raw data that Amazon Monitron is storing for you, in order to stay informed about exactly what kind of data you’re securely storing with AWS. You can get your raw data by filing a support ticket with AWS, and by giving Amazon Monitron permission to deliver your data to you. To get real time operational data for Amazon Monitron resources that can be consumed programmatically, consider exporting your data using Kinesis streams. For more information, see [Amazon Monitron Kinesis data export v2](https://docs.aws.amazon.com/Monitron/latest/user-guide/monitron-kinesis-export-v2.html). **Topics** + [Prerequisites](exporting-data-procedure.md) + [Exporting your data with CloudFormation (recommended option)](onetime-download-cflink.md) + [Exporting your data with the console](onetime-download-console.md) + [Exporting your data with CloudShell](export-shell.md) # Prerequisites To successfully export your Amazon Monitron data, the following prerequisites must be met. + You must not already have another export (of Amazon Monitron data) running in the same region. + You cannot have run another export in same region in past 24 hours. # Exporting your data with CloudFormation (recommended option) **Topics** + [Step 1: Create your Amazon S3 bucket, IAM role, and IAM policies.](#gdpr-cloudfront-makestack) + [Step 2: Note your resources](#gdpr-cloudfront-resources) + [Step 3: Create the support case](#gdpr-cloudfront-case) ## Step 1: Create your Amazon S3 bucket, IAM role, and IAM policies. 1. Sign in to your AWS account. 1. Open a new browser tab with the following URL. ``` https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/monitron-cloudformation-templates-us-east-1/monitron_manual_download.yaml&stackName=monitronexport ``` 1. On the CloudFormation page that opens, in the upper right corner, select the region in which you are using Amazon Monitron. 1. Choose **Create stack**. ![\[CloudFormation quick create stack interface with template URL, stack name, and IAM role options.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-1.png) 1. On the next page, choose the refresh icon as often as you like until the status of the stack (monitronexport) is CREATE\$1COMPLETE. ![\[CloudFormation stack details page showing monitronexport stack in CREATE_IN_PROGRESS state.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-2.png) ## Step 2: Note your resources 1. Choose the **Outputs** tab. 1. Note the value of the key `MonRoleArn`. 1. Note the value of the key `S3BucketArn`. 1. Note your account ID from the upper right corner of the page). 1. Note the region you chose in Step 1. It also now appears at the top of the page, to the left of your account ID. ![\[CloudFormation stack outputs page showing MonRoleArn and S3BucketArn with descriptions.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-3.png) ## Step 3: Create the support case 1. From your AWS console, choose the question mark icon near the upper right corner of any page, then choose **Support Center**. ![\[AWS console interface showing IAM dashboard with Support Center dropdown menu highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-support-question-mark.png) 1. On the next page, choose **Create case**. ![\[Support Center interface with Quick solutions, Active cases, and Create case button.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-4.png) 1. On the **How can we help?** page, do the following: 1. Choose **Account and billing support**. 1. Under **Service**, choose **Account**. 1. Under **Category**, choose **Compliance & Accreditations**. 1. Choose **Severity**, if that option is available to you based on your support subscription. 1. Choose **Next step: Additional information**. ![\[Support case form with Account and billing selected, and service details specified.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-5.png) 1. In **Additional information** do the following: 1. Under **Subject**, enter **Amazon Monitron data export request**. 1. In the **Description** field, enter: 1. your account ID 1. the region of the bucket you created 1. the ARN of the bucket you created (for example: "arn:aws:s3:::bucketname") 1. the ARN of the role you created (for example: "arn:aws:iam::273771705212:role/role-for-monitron") ![\[Form for Amazon Monitron data export request with fields for account and bucket details.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-6.png) 1. Choose **Next step: Solve now or contact us**. 1. In **Solve now or contact us** do the following: 1. In **Solve now**, select **Next**. ![\[Support options interface with "Solve now" and "Contact us" buttons, and recommendations.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-7.png) 1. In **Contact us**, choose your **Preferred contact language** and preferred method of contact. 1. Choose **Submit**. A confirmation screen with your case ID and details will be displayed. ![\[Contact options with language selection and choices for Web, Phone, or Chat communication.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-8.png) An AWS customer support specialist will get back to you as soon as possible. If there are any issues with the steps listed, the specialist may ask you for more information. If all the necessary information has been provided, the specialist will let you know as soon as your data has been copied to the Amazon S3 bucket that you created above. # Exporting your data with the console **Topics** + [Step 1: Setting up your Amazon S3 bucket](#gdpr-console-s3) + [Step 2: Give Amazon Monitron permission to access Amazon S3](#gdpr-console-set-policy) + [Step 3: Create the role](#gdpr-console-create-role) + [Step 4: Create the trust policy](#gdpr-console-trust-policy) + [Step 5: Create the support case](#gdpr-console-case) ## Step 1: Setting up your Amazon S3 bucket 1. Open the [Amazon S3 console](https://console.aws.amazon.com/s3/). 1. Choose **Create bucket**. ![\[Amazon S3 console interface showing Buckets section with Create bucket button highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-create-bucket.png) 1. Name your bucket and select an appropriate region. Then, at the bottom of the page, choose **Create bucket**. **Important** At this time, Amazon Monitron is only supported in three regions: US East (N. Virginia) us-east-1 EU (Ireland) eu-west-1 Asia Pacific (Sydney) ap-south-east-2 Therefore, your Amazon S3 bucket must be in one of those regions. It must also be the same region in which you are using the Amazon Monitron service. ![\[Create bucket interface showing bucket name "monitron-export-example" and AWS Region "US East (N. Virginia)".\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-create-bucket-2.png) 1. Review the rest of the options on the page, and make the appropriate choices, depending on your security needs and policies. **Important** You are responsible for taking the appropriate steps to secure your data. We strongly recommend using server-side encryption and blocking public access to your bucket. 1. Using the search box, find the bucket you just created, and then choose it. ![\[AWS S3 console showing a newly created bucket named "monitron-export-example" in the bucket list.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-choose-s3-bucket.png) 1. From the **Properties** tab, make a note of the name, ARN, and region of the bucket. ![\[S3 bucket properties showing name, region, ARN, and creation date for monitron-export-example.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-s3-properties-tab.png) ## Step 2: Give Amazon Monitron permission to access Amazon S3 1. Open the [ IAM console](https://console.aws.amazon.com/iam/) and choose **Policies**. ![\[IAM Dashboard showing resource counts, recent updates, and tools for policy management.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-9.png) 1. Choose **Create policy**. ![\[IAM Policies page with options to search, filter, and create a new policy.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-10.png) 1. Select the **JSON** tab. ![\[Policy editor interface showing JSON structure for specifying permissions in IAM.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-11.png) 1. Delete the default JSON text so that the form is empty. 1. Paste in the bucket access policy. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::bucketname" ] }, { "Action": [ "s3:PutObject", "s3:GetBucketAcl" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::bucketname/*" ] } ] } ``` ------ ![\[IAM policy editor interface showing JSON code for S3 bucket permissions.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-12.png) 1. Select **Next**. 1. On the **Review and create** page, do the following: 1. In **Policy details**, enter a **Policy name** and optional **Description**. 1. Leave the **Permissions defined in this policy** section as is. 1. In **Add tags — *optional*, you can choose to add tags to keep track of your resources.**. 1. Choose **Create policy**. ![\[Policy creation interface showing policy details, permissions, and tags sections.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-13.png) ## Step 3: Create the role 1. Open the [ IAM console](https://console.aws.amazon.com/iam/) and choose **Roles**. ![\[IAM Dashboard showing resource counts, recent updates, and available tools.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-14.png) 1. Choose **Create role**. ![\[IAM roles interface showing 116 roles and a prominent "Create role" button.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-15.png) 1. On the **Select trusted entity**, in **Trusted entity type**, choose **AWS account**. 1. In **An AWS account**, choose **This account**. You can customize additional setting using **Options**. 1. Choose **Next**. ![\[AWS account selection interface with trusted entity types and account options.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-16.png) 1. In **Add permissions**, for **Permissions policies**, search for the policy you just created in the search box, and select your policy. ![\[Add permissions interface showing search for "monitron-policy" with one matching result selected.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-17.png) 1. On the **Name, review, and create** page do the following: 1. In **Role details** enter a **Role name** and optional **Description**. 1. You can choose to ignore **Step 1: Select trusted entities** and **Step 2: Add permisions**. 1. For **Step 3: Add tags**, for **Add tags — *optional***, add optional tags to keep track of your resources. 1. Choose **Create role**. ![\[Form for creating a new role with fields for role name, description, trust policy, and permissions.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-18.png) ## Step 4: Create the trust policy 1. Search for the role you just created and choose the role. ![\[IAM Roles page showing search results for "monitron-role" with one matching role listed.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-19.png) 1. Select the **Trust relationships** tab. ![\[IAM role details page showing Trust relationships tab and Edit trust policy button.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-20.png) 1. Choose **Edit trust relationship**. ![\[Trust relationships tab with Edit trust relationship button highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-edit-trust-relationship.png) 1. Erase the default JSON text so that the form is empty. 1. Paste in the policy that allows Amazon Monitron to assume the role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": ["monitron.amazonaws.com"] }, "Action": "sts:AssumeRole" }] } ``` ------ ![\[Form for creating a new role with fields for role name, description, trust policy, and permissions.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-18.png) 1. Choose **Update Trust Policy**. ## Step 5: Create the support case 1. From your AWS console, choose the question mark icon near the upper right corner of any page, then choose **Support Center**. ![\[AWS console interface showing IAM dashboard with Support Center dropdown menu highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-support-question-mark.png) 1. On the next page, choose **Create case**. ![\[Support Center interface with Quick solutions, Active cases, and Create case button.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-4.png) 1. On the **How can we help?** page, do the following: 1. Choose **Account and billing support**. 1. Under **Service**, choose **Account**. 1. Under **Category**, choose **Compliance & Accreditations**. 1. Choose **Severity**, if that option is available to you based on your support subscription. 1. Choose **Next step: Additional information**. ![\[Support case form with Account and billing selected, and service details specified.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-5.png) 1. In **Additional information** do the following: 1. Under **Subject**, enter **Amazon Monitron data export request**. 1. In the **Description** field, enter: 1. your account ID 1. the region of the bucket you created 1. the ARN of the bucket you created (for example: "arn:aws:s3:::bucketname") 1. the ARN of the role you created (for example: "arn:aws:iam::273771705212:role/role-for-monitron") ![\[Form for Amazon Monitron data export request with fields for account and bucket details.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-6.png) 1. Choose **Next step: Solve now or contact us**. 1. In **Solve now or contact us** do the following: 1. In **Solve now**, select **Next**. ![\[Support options interface with "Solve now" and "Contact us" buttons, and recommendations.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-7.png) 1. In **Contact us**, choose your **Preferred contact language** and preferred method of contact. 1. Choose **Submit**. A confirmation screen with your case ID and details will be displayed. ![\[Contact options with language selection and choices for Web, Phone, or Chat communication.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-8.png) An AWS customer support specialist will get back to you as soon as possible. If there are any issues with the steps listed, the specialist may ask you for more information. If all the necessary information has been provided, the specialist will let you know as soon as your data has been copied to the Amazon S3 bucket that you created above. # Exporting your data with CloudShell **Topics** + [Step 1: Creating an Amazon S3 bucket (with AWS CloudShell)](#create-s3-with-shell) + [Step 2: Granting Amazon Monitron access to your Amazon S3 bucket (with AWS CloudShell)](#create-policy-with-shell) + [Step 3: Creating your support ticket](#create-support-ticket) ## Step 1: Creating an Amazon S3 bucket (with AWS CloudShell) 1. Log in to the AWS Console. 1. Open AWS CloudShell [AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) is a command-line environment that operates inside your browser. Inside AWS CloudShell, you can use the AWS Command Line Interface to launch and configure many AWS services. 1. In AWS CloudShell, enter the following command, where bucketname is the name of the bucket you are creating: ``` $ aws s3api create-bucket --bucket bucketname --region us-east-1 ``` This command creates an Amazon S3 bucket to store your raw data. You will be able to easily access your bucket from the console, and download your data at your convenience. For more information, see [Creating, configuring, and working with Amazon S3 buckets](https://docs.aws.amazon.com//AmazonS3/latest/userguide/creating-buckets-s3.html). **Important** You are responsible for taking the appropriate steps to secure your data. We strongly recommend using server-side encryption and blocking public access to your bucket. In the command above, the bucket is created in the US East (N. Virginia) Region. You can optionally specify a different Region in the request body. For more information, see [Regions, Availability Zones, and Local Zones](https://docs.aws.amazon.com//AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html). You should see output that looks something like this: ``` { "Location": "/bucketname" } ``` 1. Identify the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com//general/latest/gr/aws-arns-and-namespaces.html) of the bucket you created, which will be: ``` arn:aws:s3:::bucketname ``` ## Step 2: Granting Amazon Monitron access to your Amazon S3 bucket (with AWS CloudShell) 1. Paste the code below into a text editor, and save it as: monitron-assumes-role.json. Do not use Microsoft Word, which will add extra characters. Use a simple text editor like Notepad or TextEdit. This policy gives Amazon Monitron permission to assume the role that will allow it to access your S3 bucket. For more information, see [ Policies and permissions in IAM.](https://docs.aws.amazon.com//IAM/latest/UserGuide/access_policies.html) ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": ["monitron.amazonaws.com"] }, "Action": "sts:AssumeRole" }] } ``` ------ 1. Paste the text below into a text editor, and save it as: monitron-role-accesses-s3.json This policy will allow Amazon Monitron (using the role created above) to access your Amazon S3 bucket. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::bucketname" ] }, { "Action": [ "s3:PutObject", "s3:GetBucketAcl" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::bucketname/*" ] } ] } ``` ------ 1. In the text file you just created, replace every occurrence of *bucketname* with the name of your bucket. For example, if the name of your bucket is relentless, then your file will look like this: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::relentless" ] }, { "Action": [ "s3:PutObject", "s3:GetBucketAcl" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::relentless/*" ] } ] } ``` ------ 1. Upload both of the json files that you just created to CloudShell in the home directory. To upload a file, choose Actions from the upper right hand corner of the CloudShell console page, then choose Upload file. 1. Enter the following on the command line in CloudShell: **aws iam create-role --role-name role-for-monitron --assume-role-policy-document "cat monitron-assumes-role.json"** This command creates the role and attaches the monitron-assumes-role policy. You should see output that looks something like this: ``` { "Role": { "Path": "/", "RoleName": "role-for-monitron", "RoleId": "AROAT7PQQWN6BMTMASVPP", "Arn": "arn:aws:iam::273771705212:role/role-for-monitron", "CreateDate": "2021-07-14T02:48:15+00:00", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Sid": "", "Effect": "Allow", "Principal": { "Service": [ "monitron.amazonaws.com" ] }, "Action": "sts:AssumeRole" }] } } } ``` Take note of the ARN value for the role you just created. You will need it later. In our example, the ARN value is: `arn:aws:iam::273771705212:role/role-for-monitron` 1. Enter the following on the command line in CloudShell: ** aws iam create-policy --policy-name role-uses-bucket --policy-document "cat role-uses-bucket.json"** This command creates the monitron-role-accesses-s3 policy. You should see output that looks something like this: ``` { "Policy": { "PolicyName": "role-uses-bucket", "PolicyId": "ANPAT7PQQWN6I5KLORSDQ", "Arn": "arn:aws:iam::273771705212:policy/role-uses-bucket", "Path": "/", "DefaultVersionId": "v1", "AttachmentCount": 0, "PermissionsBoundaryUsageCount": 0, "IsAttachable": true, "CreateDate": "2021-07-14T02:19:23+00:00", "UpdateDate": "2021-07-14T02:19:23+00:00" } } ``` Take note of the ARN value for the policy that you just created. You will need it for the next step. In our example, the ARN value is: ``` arn:aws:iam::273771705212:policy/role-uses-bucket ``` 1. Enter the following on the command line in CloudShell, replacing the ARN with the ARN for your role-uses-bucket policy: ``` aws iam attach-role-policy --role-name role-for-monitron --policy-arn arn:aws:iam::273771705212:policy/role-uses-bucket ``` This command attaches the monitron-role-accesses-s3 policy to the role you just created. Now you have created and provisioned an Amazon S3 bucket, a role that Amazon Monitron can assume, a policy that will allow Amazon Monitron to assume that role, and another policy that will allow the service using that role to use your Amazon S3 bucket. You are responsible for taking the appropriate steps to secure your data. We strongly recommend using server-side encryption and blocking public access to your bucket. For more information, see [ Blocking public access](https://docs.aws.amazon.com//AmazonS3/latest/userguide/access-control-block-public-access.html). ## Step 3: Creating your support ticket 1. From your AWS console, choose the question mark icon near the upper right corner of any page, then choose **Support Center**. ![\[AWS console interface showing IAM dashboard with Support Center dropdown menu highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/gdpr-support-question-mark.png) 1. On the next page, choose **Create case**. ![\[Support Center interface with Quick solutions, Active cases, and Create case button.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-4.png) 1. On the **How can we help?** page, do the following: 1. Choose **Account and billing support**. 1. Under **Service**, choose **Account**. 1. Under **Category**, choose **Compliance & Accreditations**. 1. Choose **Severity**, if that option is available to you based on your support subscription. 1. Choose **Next step: Additional information**. ![\[Support case form with Account and billing selected, and service details specified.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-5.png) 1. In **Additional information** do the following: 1. Under **Subject**, enter **Amazon Monitron data export request**. 1. In the **Description** field, enter: 1. your account ID 1. the region of the bucket you created 1. the ARN of the bucket you created (for example: "arn:aws:s3:::bucketname") 1. the ARN of the role you created (for example: "arn:aws:iam::273771705212:role/role-for-monitron") ![\[Form for Amazon Monitron data export request with fields for account and bucket details.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-6.png) 1. Choose **Next step: Solve now or contact us**. 1. In **Solve now or contact us** do the following: 1. In **Solve now**, select **Next**. ![\[Support options interface with "Solve now" and "Contact us" buttons, and recommendations.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-7.png) 1. In **Contact us**, choose your **Preferred contact language** and preferred method of contact. 1. Choose **Submit**. A confirmation screen with your case ID and details will be displayed. ![\[Contact options with language selection and choices for Web, Phone, or Chat communication.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/s3-export-8.png) An AWS customer support specialist will get back to you as soon as possible. If there are any issues with the steps listed, the specialist may ask you for more information. If all the necessary information has been provided, the specialist will let you know as soon as your data has been copied to the Amazon S3 bucket that you created above. # Amazon Monitron Kinesis data export v1 **Note** Amazon Monitron Kinesis data export schema v1 has been deprecated. Learn more about the [v2 data export schema](monitron-kinesis-export-v2.md). You can export incoming measurement data and the corresponding inference results from Amazon Monitron and perform real-time analysis. Data export streams live data to Kinesis. **Topics** + [Exporting your data to a Kinesis stream](#exporting-stream-procedure) + [Editing live data export settings](#edit-live-export) + [Stopping a live data export](#stop-kinesis-export) + [Viewing data export errors](#viewing-kinesis-export-errors) + [Using server-side encryption for the Kinesis stream](#data-export-server-side-encryption) + [Monitoring with Amazon CloudWatch Logs](data-export-cloudwatch-logs.md) + [Storing exported data in Amazon S3](kinesis-store-S3.md) + [Processing data with Lambda](data-export-lambda.md) + [Understanding the v1 data export schema](data-export-schema.md) ## Exporting your data to a Kinesis stream 1. From your project's main page, near the bottom of the page, on the right, choose **Start live data export**. 1. Under **Select Amazon Kinesis data stream**, do one of the following: + Enter the name of an existing stream in the search box. Then skip to Step 5. + Choose **Create a new data stream**. 1. On the **Create data stream** page, under **Data stream configuration**, enter your data stream name. 1. Under Data stream capacity, choose your capacity mode: + If your data stream’s throughput requirements are unpredictable and variable, choose **On-demand**. + If you can reliably estimate the throughput requirements of your data stream, choose **Provisioned**. Then, under provisioned shards, enter the number of shards you want to create, or choose the **Shard estimator**. 1. Choose **Create data stream**. ## Editing live data export settings To edit your live data export settings: 1. Open the Amazon Monitron console. 1. Choose **Projects** from the navigation pane. 1. If you have multiple projects, choose the project for which you want to edit the export settings. 1. From the main page for your project, under **Live data export**, from the **Actions** dropdown menu, choose **Edit live data export settings**. ## Stopping a live data export 1. Open the Amazon Monitron console. 1. Choose **Projects** from the navigation pane. 1. If you have multiple projects, choose the project for which you want to edit the export settings. 1. From the main page for your project, under **Live data export**, from the **Actions** dropdown menu, choose **Stop live data export**. 1. In the pop-up window, choose **Stop**. ## Viewing data export errors To view the error messages in the CloudWatch Logs interface: + On the Amazon Monitron console, from the main page for your project, under **Live data export**, choose **CloudWatch log group**. ## Using server-side encryption for the Kinesis stream You can enable server-side encryption for your Kinesis stream before setting up Kinesis data export. However, if server-side encryption is enabled after Kinesis data export is set up, Amazon Monitron will not be able to publish to the stream. That's because Amazon Monitron will not have permission to call [kms:GenerateDataKey](https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKey.html) so that it can encrypt data sent to Kinesis. To work around this, follow the instructions under [Editing live data export settings](#edit-live-export), but without changing the configuration. This will associate the encryption you have set up with your export configuration. # Monitoring with Amazon CloudWatch Logs You can monitor Amazon Monitron live data export using Amazon CloudWatch Logs. When a measurement fails to export, Amazon Monitron will send a log event to your CloudWatch Logs. You can also set up a metric filter on the error log to generate metrics and set up alarms. An alarm can watch for certain thresholds and send notifications or take actions when those thresholds are met. For more information, see [the CloudWatch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). Amazon Monitron sends log events to the /aws/monitron/data-export/\$1HASH\$1ID\$1 log group. The log event has the following JSON format: ``` { "assetDisplayName": "string", "destination": "string", "errorCode": "string", "errorMessage": "string", "eventId": "string", "positionDisplayName": "string", "projectDisplayName": "string", "projectName": "string", "sensorId": "string", "siteDisplayName": "string", "timestamp": "string" } ``` assetDisplayName + The asset name displayed in the App + Type: String destination + The ARN of the Kinesis data stream + Type: String + Pattern: arn:aws:kinesis:\$1\$1REGION\$1\$1:\$1\$1AWS\$1ACCOUNT\$1ID\$1\$1:stream/\$1\$1STREAM\$1NAME\$1\$1 errorCode + The error code + Type: String + Valid Values: `INTERNAL_SEVER_ERROR | KINESIS_RESOURCE_NOT_FOUND | KINESIS_PROVISIONED_THROUGHPUT_EXCEEDED | KMS_ACCESS_DENIED | KMS_NOT_FOUND | KMS_DISABLED | KMS_INVALID_STATE | KMS_THROTTLING` errorMessage + The detailed error message + Type: String eventId + The unique event ID corresponding to each measurement export + Type: String positionDisplayName + The sensor position name displayed in the App + Type: String sensorId + The physical ID of the sensor from which the measurement is sent + Type: String siteDisplayName + The site name displayed in the App + Type: String timestamp + The timestamp when the measurement is received by Amazon Monitron service in UTC + Type: String + Pattern: yyyy-mm-dd hh:mm:ss.SSS # Storing exported data in Amazon S3 **Topics** + [Using a predefined CloudFormation template](#kinesis-cloudfront-makestack) + [Configuring Kinesis manually in the console](#kinesis-configure-console) ## Using a predefined CloudFormation template Amazon Monitron provides a predefined AWS CloudFormation template to help quickly set up the Firehose to deliver data from a Kinesis data stream to the Amazon S3 bucket. This template enables dynamic partitioning and the Amazon S3 objects delivered will use the following key format recommended by Amazon Monitron: `/project={projectName}/site={siteName}/time={yyyy-mm-dd 00:00:00}/{filename}` 1. Sign into your AWS account. 1. Open a new browser tab with the following URL: ``` https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/monitron-cloudformation-templates-us-east-1/monitron_kinesis_data_export.yaml&stackName=monitron-kinesis-live-data-export ``` 1. On the CloudFormation page that opens, in the upper right corner, select the region in which you are using Amazon Monitron. 1. By default, the template will create a new Kinesis data stream and S3 bucket along with other resources needed to deliver data to Amazon S3. You can change the parameters to use existing resources. 1. Check the box that says *I acknowledge that AWS CloudFormation might create IAM resources.* 1. Choose **Create stack**. 1. On the next page, choose the refresh icon as often as you like until the status of the stack is CREATE\$1COMPLETE. ## Configuring Kinesis manually in the console 1. Sign in to the AWS Management Console and open the Kinesis console at https://console.aws.amazon.com/kinesis. 1. Choose **Delivery streams** in the navigation pane. 1. Choose **Create delivery stream**. 1. For Source, select **Amazon Kinesis Data Streams**. 1. For Destination, select **Amazon S3**. 1. Under **Source settings, Kinesis data stream**, enter the ARN of your Kinesis data stream. 1. Under **delivery stream name**, enter the name of your Kinesis data stream. 1. Under **Desination settings**, choose an Amazon S3 bucket or enter a bucket URI. 1. (optional) Enable dynamic partitioning using inline parsing for JSON. This option is appropriate if you want to partition streaming measurement data based on source information and timestamp. For example: + Choose **Enabled** for **Dynamic partitioning**. + Choose **Enabled** for **New line delimiter**. + Choose **Enabled** for **Inline parsing for JSON**. + Under **Dynamic partitioning keys**, add: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/kinesis-store-S3.html) 1. Choose **Apply dynamic partitioning keys** and confirm the generated Amazon S3 bucket prefix is `!{partitionKeyFromQuery:project}/!{partitionKeyFromQuery:site}/!{partitionKeyFromQuery:time}/`. 1. In Amazon S3, objects will use the following key format: `/project={projectName}/site={siteName}/time={yyyy-mm-dd 00:00:00}/{filename}`. 1. Choose **Create delivery stream**. 1. (optional) Use a more granular path. If you chose a dynamic partition, use the preceeding Amazon S3 key format if you plan to use AWS Glue and Athena to query the data. You can also choose a finer key format, but the Amazon Athena query will not be efficient. Here is an example of setting up a finer Amazon S3 key path. Under **Dynamic partitioning keys**, add: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/kinesis-store-S3.html) In Amazon S3, objects will use the following key format: ` /project={projectName}/site={siteName}/asset={assetName}/position={positionName}/sensor={sensorId}/date={yyyy-mm-dd}/time={HH:MM:SS}/{filename}` # Processing data with Lambda **Topics** + [Step 1: Create the [IAM role](https://docs.aws.amazon.com//lambda/latest/dg/lambda-intro-execution-role.html) that gives your function permission to access AWS resources](#create-iam-role) + [Step 2: Create the Lambda function](#create-lambda-function) + [Step 3: Configure the Lambda function](#configure-lambda-function) + [Step 4: Enable Kinesis trigger in AWS Lambda console](#enable-lambda-trigger) ## Step 1: Create the [IAM role](https://docs.aws.amazon.com//lambda/latest/dg/lambda-intro-execution-role.html) that gives your function permission to access AWS resources 1. Open the [roles page](https://console.aws.amazon.com/iam/home?#/roles) in the IAM console. 1. Choose **Create role**. 1. Create a role with the following properties. + Trusted entity: Lambda + Permissions: AWSLambdaKinesisExecutionRole (and AWSKeyManagementServicePowerUser if the Kinesis stream is encrypted) + Role name: lambda-kinesis-role ![\[IAM role creation interface showing name, review, and create steps with role details and permissions.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-create-role.png) ## Step 2: Create the Lambda function 1. Open the **Functions** page in the Lambda console. 1. Choose **Create function**. 1. Choose **Use a blueprint**. 1. In the **Blueprints** search bar, search and choose **kinesis-process-record (nodejs)** or **kinesis-process-record-python**. 1. Choose **Configure**. ![\[Create function interface with options to author from scratch, use a blueprint, or select container image.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-create-function.png) ## Step 3: Configure the Lambda function 1. Choose **Function name** 1. Choose the role created in the first step as the **Execution role**. 1. Configure Kinesis trigger. 1. Choose your Kinesis stream. 1. Click **Create function**. ![\[Lambda function configuration form with basic information and Kinesis trigger settings.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-kinesis-trigger.png) ## Step 4: Enable Kinesis trigger in AWS Lambda console 1. On the **Configuration** tab, choose **Triggers**. 1. Check the box next to the name of the Kinesis stream and choose **Enable**. ![\[Lambda function configuration page with Triggers tab and Kinesis stream trigger highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/kinesis-process-record-lambda.png) The blueprint used in this example only consumes log data from the selected stream. You can further edit Lambda function code later to complete a more complicated task. # Understanding the v1 data export schema **Note** Amazon Monitron Kinesis data export schema v1 has been deprecated. Learn more about the [v2 data export schema](monitron-kinesis-export-v2.md). Each measurement data and its corresponding inference result are exported as one Kinesis data stream record in JSON format. **Topics** + [v1 schema format](#data-export-schema-format) + [v1 schema parameters](#data-export-schema-parameters) ## v1 schema format ``` { "timestamp": "string", "eventId": "string", "version": "string", "projectDisplayName": "string", "siteDisplayName": "string", "assetDisplayName": "string", "sensorPositionDisplayName": "string", "sensor": { "physicalId": "string", "rssi": number }, "gateway": { "physicalId": "string" }, "measurement": { "features": { "acceleration": { "band0To6000Hz": { "xAxis": { "rms": number }, "yAxis": { "rms": number }, "zAxis": { "rms": number } }, "band10To1000Hz": { "resultantVector": { "absMax": number, "absMin": number, "crestFactor": number, "rms": number }, "xAxis": { "rms": number }, "yAxis": { "rms": number }, "zAxis": { "rms": number } } }, "temperature": number, "velocity": { "band10To1000Hz": { "resultantVector": { "absMax": number, "absMin": number, "crestFactor": number, "rms": number }, "xAxis": { "rms": number }, "yAxis": { "rms": number }, "zAxis": { "rms": number } } } }, "sequenceNo": number }, "models": { "temperatureML": { "persistentClassificationOutput": "string", "pointwiseClassificationOutput": "string" }, "vibrationISO": { "isoClass": "string", "mutedThreshold": "string", "persistentClassificationOutput": "string", "pointwiseClassificationOutput": "string" }, "vibrationML": { "persistentClassificationOutput": "string", "pointwiseClassificationOutput": "string" } }, "assetState": { "newState": "string", "previousState": "string" } } ``` ## v1 schema parameters timestamp + The timestamp when the measurement is received by Monitron service in UTC + Type: String + Pattern: yyyy-mm-dd hh:mm:ss.SSS eventId + The unique data export event ID assigned for each measurement. Can be used to deduplicate the Kinesis stream records received. + Type: String version + Schema version + Type: String + Current Value: 1.0 projectDisplayName + The project name displayed in the App and console + Type: String siteDisplayName + The site name displayed in the App + Type: String assetDisplayName + The asset name displayed in the App + Type: String sensorPositionDisplayName + The sensor position name displayed in the App + Type: String sensor.physicalId + The physical ID of the sensor from which the measurement is sent + Type: String sensor.rssi + The sensor bluetooth received signal strength indicator value + Type: Number + Unit: dBm gateway.physicalId + The physical ID of the gateway used to transmit data to Amazon Monitron service + Type: String measurement.features.acceleration.band0To6000Hz.xAxis.rms + The root mean square of the acceleration observed in the frequency band 0–6000 Hz in the x axis + Type: Number + Unit: m/s^2 measurement.features.acceleration.band0To6000Hz.yAxis.rms + The root mean square of the acceleration observed in the frequency band 0–6000 Hz in the y axis + Type: Number + Unit: m/s^2 measurement.features.acceleration.band0To6000Hz.zAxis.rms + The root mean square of the acceleration observed in the frequency band 0–6000 Hz in the y axis + Type: Number + Unit: m/s^2 measurement.features.acceleration.band10To1000Hz.resultantVector.absMax + The absolute maximum acceleration observed in the frequency band 10–1000 Hz + Type: Number + Unit: m/s^2 measurement.features.acceleration.band10To1000Hz.resultantVector.absMin + The absolute minimum acceleration observed in the frequency band 10–1000 Hz + Type: Number + Unit: m/s^2 measurement.features.acceleration.band10To1000Hz.resultantVector.crestFactor + The acceleration crest factor observed in the frequency band 10–1000 Hz + Type: Number measurement.features.acceleration.band10To1000Hz.resultantVector.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz + Type: Number + m/s^2 measurement.features.acceleration.band10To1000Hz.xAxis.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz in the x axis + Type: Number + m/s^2 measurement.features.acceleration.band10To1000Hz.yAxis.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz in the y axis + Type: Number + m/s^2 measurement.features.acceleration.band10To1000Hz.zAxis.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz in the z axis + Type: Number + m/s^2 measurement.features.temperature + The temperature observed + Type: Number + °C/degC measurement.features.velocity.band10To1000Hz.resultantVector.absMax + The absolute maximum velocity observed in the frequency band 10–1000 Hz + Type: Number + mm/s measurement.features.velocity.band10To1000Hz.resultantVector.absMin + The absolute minimum velocity observed in the frequency band 10–1000 Hz + Type: Number + mm/s measurement.features.velocity.band10To1000Hz.resultantVector.crestFactor + The velocity crest factor observed in the frequency band 10–1000 Hz + Type: Number measurement.features.velocity.band10To1000Hz.resultantVector.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz + Type: Number + mm/s measurement.features.velocity.band10To1000Hz.xAxis.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz in the x axis + Type: Number + mm/s measurement.features.velocity.band10To1000Hz.yAxis.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz in the y axis + Type: Number + mm/s measurement.features.velocity.band10To1000Hz.zAxis.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz in the z axis + Type: Number + mm/s measurement.sequenceNo + The measurement sequence number + Type: Number models.temperatureML.persistentClassificationOutput + The persistent classification output from the machine learning based temperature model + Type: Number + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM` models.temperatureML.pointwiseClassificationOutput + The point–wise classification output from the machine learning based temperature model + Type: String + Valid Values: `UNKNOWN | INITIALIZING | HEALTHY | WARNING | ALARM` models.vibrationISO.isoClass + The ISO 20816 class (a standard for measurement and evaluation of machine vibration) used by the ISO based vibration model + Type: String + Valid Values: `CLASS1 | CLASS2 | CLASS3 | CLASS4 | FAN_BV2` models.vibrationISO.mutedThreshold + The threshold to mute the notification from the ISO based vibration model + Type: String + Valid Values: `WARNING | ALARM` models.vibrationISO.persistentClassificationOutput + The persistent classification output from the ISO based vibration model + Type: String + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM` models.vibrationISO.pointwiseClassificationOutput + The point–wise classification output from the the ISO based vibration model + Type: String + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM | MUTED_WARNING | MUTED_ALARM` models.vibrationML.persistentClassificationOutput + The persistent classification output from the machine learning based vibration model + Type: String + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM` models.vibrationML.pointwiseClassificationOutput + The point–wise classification output from the machine learning based vibration model + Type: String + Valid Values: `UNKNOWN | INITIALIZING | HEALTHY | WARNING | ALARM` assetState.newState + The machine status after processing the measurement + Type: String + Valid Values: `UNKNOWN | HEALTHY | NEEDS_MAINTENANCE | WARNING | ALARM` assetState.previousState + The machine status before processing the measurement + Type: String + Valid Values: `UNKNOWN | HEALTHY | NEEDS_MAINTENANCE | WARNING | ALARM` # Amazon Monitron Kinesis data export v2 You can export incoming measurement data and the corresponding inference results from Amazon Monitron and perform real-time analysis. Data export streams live data to Kinesis. **Topics** + [Exporting your data to a Kinesis stream](#exporting-stream-procedure-v2) + [Editing live data export settings](#edit-live-export-v2) + [Stopping a live data export](#stop-kinesis-export-v2) + [Viewing data export errors](#viewing-kinesis-export-errors-v2) + [Using server-side encryption for the Kinesis stream](#data-export-server-side-encryption-v2) + [Monitoring with Amazon CloudWatch Logs](data-export-cloudwatch-logs-v2.md) + [Storing exported data in Amazon S3](kinesis-store-S3-v2.md) + [Processing data with Lambda](data-export-lambda-v2.md) + [Understanding the v2 data export schema](data-export-schema-v2.md) + [Migration from Kinesis v1 to v2](migration-from-v1-to-v2.md) ## Exporting your data to a Kinesis stream 1. From your project's main page, near the bottom of the page, on the right, choose **Start live data export**. 1. Under **Select Kinesis data stream**, do one of the following: + Enter the name of an existing stream in the search box. Then skip to Step 5. + Choose **Create a new data stream**. 1. On the **Create data stream** page, under **Data stream configuration**, enter your data stream name. 1. Under Data stream capacity, choose your capacity mode: + If your data stream’s throughput requirements are unpredictable and variable, choose **On-demand**. + If you can reliably estimate the throughput requirements of your data stream, choose **Provisioned**. Then, under provisioned shards, enter the number of shards you want to create, or choose the **Shard estimator**. 1. Choose **Create data stream**. ## Editing live data export settings To edit your live data export settings: 1. Open the Amazon Monitron console. 1. Choose **Projects** from the navigation pane. 1. If you have multiple projects, choose the project for which you want to edit the export settings. 1. From the main page for your project, under **Live data export**, from the **Actions** dropdown menu, choose **Edit live data export settings**. ## Stopping a live data export 1. Open the Amazon Monitron console. 1. Choose **Projects** from the navigation pane. 1. If you have multiple projects, choose the project for which you want to edit the export settings. 1. From the main page for your project, under **Live data export**, from the **Actions** dropdown menu, choose **Stop live data export**. 1. In the pop-up window, choose **Stop**. ## Viewing data export errors To view the error messages in the CloudWatch Logs interface: + On the Amazon Monitron console, from the main page for your project, under **Live data export**, choose **CloudWatch log group**. ## Using server-side encryption for the Kinesis stream You can enable server-side encryption for your Kinesis stream before setting up Kinesis data export. However, if server-side encryption is enabled after Kinesis data export is set up, Amazon Monitron will not be able to publish to the stream. That's because Amazon Monitron will not have permission to call [kms:GenerateDataKey](https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKey.html) so that it can encrypt data sent to Kinesis. To work around this, follow the instructions under [Editing live data export settings](#edit-live-export-v2), but without changing the configuration. This will associate the encryption you have set up with your export configuration. # Monitoring with Amazon CloudWatch Logs You can monitor Amazon Monitron live data export using Amazon CloudWatch Logs. When a measurement fails to export, Amazon Monitron will send a log event to your CloudWatch Logs. You can also set up a metric filter on the error log to generate metrics and set up alarms. An alarm can watch for certain thresholds and send notifications or take actions when those thresholds are met. For more information, see [the CloudWatch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). Amazon Monitron sends log events to the /aws/monitron/data-export/\$1HASH\$1ID\$1 log group. The log event has the following JSON format: ``` { "assetName": "string", "destination": "string", "errorCode": "string", "errorMessage": "string", "eventId": "string", "eventType": "string", "positionName": "string", "projectName": "string", "projectId": "string", "sensorId": "string", "gatewayId": "string", "siteName": "string", "timestamp": "string" } ``` assetName + The asset name displayed in the app + Type: String destination + The ARN of the Kinesis data stream + Type: String + Pattern: arn:aws:kinesis:\$1\$1REGION\$1\$1:\$1\$1AWS\$1ACCOUNT\$1ID\$1\$1:stream/\$1\$1STREAM\$1NAME\$1\$1 errorCode + The error code + Type: String + Valid Values: `INTERNAL_SEVER_ERROR | KINESIS_RESOURCE_NOT_FOUND | KINESIS_PROVISIONED_THROUGHPUT_EXCEEDED | KMS_ACCESS_DENIED | KMS_NOT_FOUND | KMS_DISABLED | KMS_INVALID_STATE | KMS_THROTTLING` errorMessage + The detailed error message + Type: String eventId + The unique event ID corresponding to each measurement export + Type: String eventType + The current event type + Type: String + Valid values: `measurement` ` | gatewayConnected` ` | gatewayDisconnected` ` | sensorConnected` ` | sensorDisconnected` ` | assetStateTransition` positionName + The sensor position name displayed in the app + Type: String projectName + The project name displayed in the app and console + Type: String projectID + The unique project ID corresponding to the Amazon Monitron project + Type: String sensorID + The physical ID of the sensor from which the measurement is sent + Type: String gatewayID + The physical ID of the gateway used to transmit data to the Amazon Monitron service + Type: String siteName + The site name displayed in the app + Type: String timestamp + The timestamp when the measurement is received by the Amazon Monitron service in UTC + Type: String + Pattern: yyyy-mm-dd hh:mm:ss.SSS # Storing exported data in Amazon S3 If you want to store your exported data in Amazon S3, use the following procedure. **Topics** + [Configuring Kinesis manually in the console](#kinesis-configure-console-v2) ## Configuring Kinesis manually in the console 1. Sign in to the AWS Management Console and open the Kinesis console at https://console.aws.amazon.com/kinesis. 1. Choose **Delivery streams** in the navigation pane. 1. Choose **Create delivery stream**. 1. For Source, select **Amazon Kinesis Data Streams**. 1. For Destination, select **Amazon S3**. 1. Under **Source settings, Kinesis data stream**, enter the ARN of your Kinesis data stream. 1. Under **delivery stream name**, enter the name of your Kinesis data stream. 1. Under **Desination settings**, choose an Amazon S3 bucket or enter a bucket URI. 1. (optional) Enable dynamic partitioning using inline parsing for JSON. This option is appropriate if you want to partition streaming measurement data based on source information and timestamp. For example: + Choose **Enabled** for **Dynamic partitioning**. + Choose **Enabled** for **New line delimiter**. + Choose **Enabled** for **Inline parsing for JSON**. + Under **Dynamic partitioning keys**, add: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/kinesis-store-S3-v2.html) 1. Choose **Apply dynamic partitioning keys** and confirm the generated Amazon S3 bucket prefix is `!{partitionKeyFromQuery:project}/!{partitionKeyFromQuery:site}/!{partitionKeyFromQuery:time}/`. 1. In Amazon S3, objects will use the following key format: `/project={projectName}/site={siteName}/time={yyyy-mm-dd 00:00:00}/{filename}`. 1. Choose **Create delivery stream**. # Processing data with Lambda **Topics** + [Step 1: Create the [IAM role](https://docs.aws.amazon.com//lambda/latest/dg/lambda-intro-execution-role.html) that gives your function permission to access AWS resources](#data-export-lambda-v2-1) + [Step 2: Create the Lambda function](#create-lambda-function-v2) + [Step 3: Configure the Lambda function](#configure-lambda-function-v2) + [Step 4: Enable Kinesis trigger in AWS Lambda console](#configure-kinesis-trigger-v2) ## Step 1: Create the [IAM role](https://docs.aws.amazon.com//lambda/latest/dg/lambda-intro-execution-role.html) that gives your function permission to access AWS resources 1. Open the [roles page](https://console.aws.amazon.com/iam/home?#/roles) in the IAM console. 1. Choose **Create role**. 1. On the **Select trusted entity** page, do the following: + In **Trusted entity type**, choose **AWS service**. + In **Use case**, for **Service or use case** choose **Lambda**. + Choose **Next**. ![\[IAM role creation interface showing trusted entity selection with AWS service option chosen.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-role-1.png) 1. In the **Add permissions** page, do the following: + In **Permissions policies**, choose AWSLambdaKinesisExecutionRole (and AWSKeyManagementServicePowerUser if the Kinesis stream is encrypted). + Leave the configurations in **Set permissions boundary** as is. + Choose **Next**. ![\[Add permissions interface showing AWSLambdaKinesisExecutionRole policy selected for a new role.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-role-2.png) 1. In the **Name, review, and create** page, do the following: + In **Role details**, for **Role name**, enter a name for your role. For example *lambda-kinesis-role*. You can also choose to add an optional **Description**. + Leave the settings for **Step 1: Select trusted entities** and **Step 2: Add permissions** as is. You can choost to add tags in **Step 3: Add tags** to keep track of your resources. ![\[IAM role creation interface showing name, review, and create steps with role details and permissions.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-create-role.png) 1. Select **Create role**. ## Step 2: Create the Lambda function 1. Open the **Functions** page in the Lambda console. 1. Choose **Create function**. 1. Choose **Use a blueprint**. 1. In the **Blueprints** search bar, search and choose **kinesis-process-record (nodejs)** or **kinesis-process-record-python**. 1. Choose **Configure**. ![\[Create function interface with options to author from scratch, use a blueprint, or select container image.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-create-function.png) ## Step 3: Configure the Lambda function 1. Choose **Function name** 1. Choose the role created in the first step as the **Execution role**. 1. Configure Kinesis trigger. 1. Choose your Kinesis stream. 1. Click **Create function**. ![\[Lambda function configuration form with basic information and Kinesis trigger settings.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/lambda-kinesis-trigger.png) ## Step 4: Enable Kinesis trigger in AWS Lambda console 1. On the **Configuration** tab, choose **Triggers**. 1. Check the box next to the name of the Kinesis stream and choose **Enable**. ![\[Lambda function configuration page with Triggers tab and Kinesis stream trigger highlighted.\]](http://docs.aws.amazon.com/Monitron/latest/user-guide/images/kinesis-process-record-lambda.png) The blueprint used in this example only consumes log data from the selected stream. You can further edit Lambda function code later to complete a more complicated task. # Understanding the v2 data export schema Each measurement data, its corresponding inference result, gateway connect/disconnect, and sensor connect/disconnect events are exported as one Kinesis data stream record in JSON format. **Topics** + [v2 schema format](#data-export-schema-format-v2) + [v2 schema parameters](#data-export-schema-parameters-v2) ## v2 schema format ``` { "timestamp": "string", "eventId": "string", "version": "2.0", "accountId": "string", "projectName": "string", "projectId": "string", "eventType": "measurement|gatewayConnected|gatewayDisconnected|sensorConnected|sensorDisconnected|assetStateTransition", // measurement "eventPayload": { "siteName": "string", "assetName": "string", "positionName": "string", "companyName": "string", "geoLocation": { "latitude": number, "longitude": number }, "address": "string", "serialNumber": "string", "make": "string", "model": "string", "assetPositionURL": "string", "sensor": { "physicalId": "string", "rssi": number }, "gateway": { "physicalId": "string" }, "sequenceNo": number, "features": { "acceleration": { "band0To6000Hz": { "xAxis": { "rms": number }, "yAxis": { "rms": number }, "zAxis": { "rms": number } }, "band10To1000Hz": { "totalVibration": { "absMax": number, "absMin": number, "crestFactor": number, "rms": number }, "xAxis": { "rms": number }, "yAxis": { "rms": number }, "zAxis": { "rms": number } } }, "velocity": { "band10To1000Hz": { "totalVibration": { "absMax": number, "absMin": number, "crestFactor": number, "rms": number }, "xAxis": { "rms": number }, "yAxis": { "rms": number }, "zAxis": { "rms": number } } }, "temperature": number } "models": { "temperatureML": { "previousPersistentClassificationOutput": "string", "persistentClassificationOutput": "string", "pointwiseClassificationOutput": "string" }, "vibrationISO": { "isoClass": "string", "mutedThreshold": "string", "previousPersistentClassificationOutput": "string", "persistentClassificationOutput": "string", "pointwiseClassificationOutput": "string" }, "vibrationML": { "previousPersistentClassificationOutput": "string", "persistentClassificationOutput": "string", "pointwiseClassificationOutput": "string" } }, "assetPositionId": "string" } // sensorConnected "eventPayload": { "siteName": "string", "assetName": "string", "positionName": "string", "companyName": "string", "geoLocation": { "latitude": number, "longitude": number }, "address": "string", "serialNumber": "string", "make": "string", "model": "string", "assetPositionURL": "string", "sensor": { "physicalId": "string" }, "assetPositionId": "string" } // sensorDisconnected "eventPayload": { "siteName": "string", "assetName": "string", "positionName": "string", "companyName": "string", "geoLocation": { "latitude": number, "longitude": number }, "address": "string", "serialNumber": "string", "make": "string", "model": "string", "assetPositionURL": "string", "sensor": { "physicalId": "string" }, "assetPositionId": "string" } // gatewayConnected "eventPayload": { "siteName": "string", "gatewayName": "string", "gatewayListURL": "string", "companyName": "string", "geoLocation": { "latitude": number, "longitude": number }, "address": "string", "gateway": { "physicalId": "string" } } // gatewayDisconnected "eventPayload": { "siteName": "string", "gatewayName": "string", "gatewayListURL": "string", "companyName": "string", "geoLocation": { "latitude": number, "longitude": number }, "address": "string", "gateway": { "physicalId": "string" } } // assetStateTransition "eventPayload": { "siteName": "string", "assetName": "string", "positionName": "string", "companyName": "string", "geoLocation": { "latitude": number, "longitude": number }, "address": "string", "serialNumber": "string", "make": "string", "model": "string", "assetPositionURL": "string", "sensor": { "physicalId": "string" }, "assetTransitionType": "measurement|userInput", "assetState": { "newState": "string", "previousState": "string" }, "closureCode": { "failureMode": "string", "failureCause": "string", "actionTaken": "string", "resolvedModels": list<"string"> }, "assetPositionId": "string" } } ``` ## v2 schema parameters The Amazon Monitron Kinesis data export schema v2 includes the following schema parameters. Some parameters are updates from v1 and some are unique to v2. For example, `siteName` was a first-level parameter in v1. In v2, it is a second-level parameter that can be found under the `eventPayload` entity. timestamp + The timestamp when the measurement is received by Amazon Monitron service in UTC + Type: String + Pattern: yyyy-mm-dd hh:mm:ss.SSS eventId + The unique data export event ID assigned for each measurement. Can be used to deduplicate the Kinesis stream records received. + Type: String version + Schema version + Type: String + Value: 1.0 or 2.0 accountId + The 12-digit AWS account ID for your Monitron project + Type: String projectName The project name displayed in the app and console. Type: String projectId The unique ID of your Amazon Monitron project. Type: String eventType + The current event stream. Each event type will have a dedicated `eventPayload` format. + Type: String + Possible values: `measurement`, `gatewayConnected`, `gatewayDisconnected`, `sensorConnected`, `sensorDisconnected`, `assetStateTransition`. **`eventType: measurement`** eventPayload.features.acceleration.band0To6000Hz.xAxis.rms + The root mean square of the acceleration observed in the frequency band 0–6000 Hz in the x axis + Type: Number + Unit: m/s^2 eventPayload.features.acceleration.band0To6000Hz.yAxis.rms + The root mean square of the acceleration observed in the frequency band 0–6000 Hz in the y axis + Type: Number + Unit: m/s^2 eventPayload.features.acceleration.band0To6000Hz.zAxis.rms + The root mean square of the acceleration observed in the frequency band 0–6000 Hz in the z axis + Type: Number + Unit: m/s^2 eventPayload.features.acceleration.band10To1000Hz.resultantVector.absMax + The absolute maximum acceleration observed in the frequency band 10–1000 Hz + Type: Number + Unit: m/s^2 eventPayload.features.acceleration.band10To1000Hz.resultantVector.absMin + The absolute minimum acceleration observed in the frequency band 10–1000 Hz + Type: Number + Unit: m/s^2 eventPayload.features.acceleration.band10To1000Hz.resultantVector.crestFactor + The acceleration crest factor observed in the frequency band 10–1000 Hz + Type: Number eventPayload.features.acceleration.band10To1000Hz.resultantVector.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz + Type: Number + m/s^2 eventPayload.features.acceleration.band10To1000Hz.xAxis.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz in the x axis + Type: Number + m/s^2 eventPayload.features.acceleration.band10To1000Hz.yAxis.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz in the y axis + Type: Number + m/s^2 eventPayload.features.acceleration.band10To1000Hz.zAxis.rms + The root mean square of the acceleration observed in the frequency band 10–1000 Hz in the z axis + Type: Number + m/s^2 eventPayload.features.temperature + The temperature observed + Type: Number + °C/degC eventPayload.features.velocity.band10To1000Hz.resultantVector.absMax + The absolute maximum velocity observed in the frequency band 10–1000 Hz + Type: Number + mm/s eventPayload.features.velocity.band10To1000Hz.resultantVector.absMin + The absolute minimum velocity observed in the frequency band 10–1000 Hz + Type: Number + mm/s eventPayload.features.velocity.band10To1000Hz.resultantVector.crestFactor + The velocity crest factor observed in the frequency band 10–1000 Hz + Type: Number eventPayload.features.velocity.band10To1000Hz.resultantVector.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz + Type: Number + mm/s eventPayload.features.velocity.band10To1000Hz.xAxis.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz in the x axis + Type: Number + mm/s eventPayload.features.velocity.band10To1000Hz.yAxis.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz in the y axis + Type: Number + mm/s eventPayload.features.velocity.band10To1000Hz.zAxis.rms + The root mean square of the velocity observed in the frequency band 10–1000 Hz in the z axis + Type: Number + mm/s eventPayload.sequenceNo + The measurement sequence number + Type: Number eventPayload.assetPositionId + The identifier of the sensor position for which the measurement is sent. + Type: String eventPayload.companyName + The name of the company using the asset. + Type: String eventPayload.geoLocation.latitude + The latitude of the site's physical location. + Type: Number eventPayload.geoLocation.longitude + The longitude of the site's physical location. + Type: Number eventPayload.address + The address of the site. + Type: String eventPayload.serialNumber + The serial number of the asset. + Type: String eventPayload.make + The make of the asset. + Type: String eventPayload.model + The model of the asset. + Type: String `eventType: sensorConnected` siteName + The site name displayed in the app + Type: String assetName + The asset name displayed in the app + Type: String positionName + The sensor position name displayed in the app + Type: String assetPositionURL + The sensor URL displayed in the app + Type: String physicalID + The physical ID of the sensor from which the measurement is sent + Type: String eventPayload.assetPositionId + The identifier of the sensor position whose state changed. + Type: String eventPayload.companyName + The name of the company using the asset. + Type: String eventPayload.geoLocation.latitude + The latitude of the site's physical location. + Type: Number eventPayload.geoLocation.longitude + The longitude of the site's physical location. + Type: Number eventPayload.address + The address of the site. + Type: String eventPayload.serialNumber + The serial number of the asset. + Type: String eventPayload.make + The make of the asset. + Type: String eventPayload.model + The model of the asset. + Type: String `eventType: sensorDisconnected` siteName + The site name displayed in the app + Type: String assetName + The asset name displayed in the app + Type: String positionName + The sensor position name displayed in the app + Type: String assetPositionURL + The sensor URL displayed in the app + Type: String physicalID + The physical ID of the sensor from which the measurement is sent + Type: String eventPayload.assetPositionId + The identifier of the sensor position whose state changed. + Type: String eventPayload.companyName + The name of the company using the asset. + Type: String eventPayload.geoLocation.latitude + The latitude of the site's physical location. + Type: Number eventPayload.geoLocation.longitude + The longitude of the site's physical location. + Type: Number eventPayload.address + The address of the site. + Type: String eventPayload.serialNumber + The serial number of the asset. + Type: String eventPayload.make + The make of the asset. + Type: String eventPayload.model + The model of the asset. + Type: String `eventType: gatewayConnected` eventPayload.siteName + The site name displayed in the app + Type: String eventPayload.gatewayName + The name of the gateway as displayed in the app + Type: String eventPayload.gatewayListURL + The gateway URL displayed in the app + Type: String eventPayload.gateway.physicalID + The physical ID of the gateway just connected to transmit data to the Amazon Monitron service + Type: String eventPayload.companyName + The name of the company using the asset. + Type: String eventPayload.geoLocation.latitude + The latitude of the site's physical location. + Type: Number eventPayload.geoLocation.longitude + The longitude of the site's physical location. + Type: Number eventPayload.address + The address of the site. + Type: String `eventType: gatewayDisconnected` siteName + The site name displayed in the app + Type: String gatewayName + The name of the gateway as displayed in the app + Type: String gatewayListURL + The gateway URL displayed in the app + Type: String physicalID + The physical ID of the gateway just connected to transmit data to the Amazon Monitron service + Type: String eventPayload.companyName + The name of the company using the asset. + Type: String eventPayload.geoLocation.latitude + The latitude of the site's physical location. + Type: Number eventPayload.geoLocation.longitude + The longitude of the site's physical location. + Type: Number eventPayload.address + The address of the site. + Type: String `eventType: assetStateTransition` eventPayload.siteName + The site name displayed in the app + Type: String eventPayload.assetName + The asset name displayed in the app + Type: String eventPayload.positionName + The sensor position name displayed in the app + Type: String eventPayload.assetPositionURL + The sensor URL displayed in the app + Type: String eventPayload.sensor.physicalID + The physical ID of the sensor from which the measurement is sent + Type: String eventPayload.assetTransitionType + The reason behind the asset state transition + Type: String + Possible values: `measurement` or `userInput` eventPayload.assetState.newState + The new state of the asset + Type: String eventPayload.assetState.previousState + The previous state of the asset + Type: String eventPayload.closureCode.failureMode + The failure mode selected by the user when acknowledging this failure + Type: String + Possible values: `NO_ISSUE` \$1 `BLOCKAGE` \$1 `CAVITATION` \$1 `CORROSION` \$1 `DEPOSIT` \$1 `IMBALANCE` \$1 `LUBRICATION` \$1 `MISALIGNMENT` \$1 `OTHER` \$1 `RESONANCE` \$1 `ROTATING_LOOSENESS` \$1 `STRUCTURAL_LOOSENESS` \$1 `TRANSMITTED_FAULT` \$1 `UNDETERMINED` eventPayload.closureCode.failureCause + The cause of the failure as selected by the user in the app dropdown when acknowledging a failure. + Type: String + Possible values: `ADMINISTRATION` \$1 `DESIGN` \$1 `FABRICATION` \$1 `MAINTENANCE` \$1 `OPERATION` \$1 `OTHER` \$1 `QUALITY` \$1 `UNDETERMINED` \$1 `WEAR` eventPayload.closureCode.actionTaken + The action taken when closing this anomaly, as selected by the user in the app dropdown. + Type: String + Possible values: `ADJUST` \$1 `CLEAN` \$1 `LUBRICATE` \$1 `MODIFY` \$1 `NO_ACTION` \$1 `OTHER` \$1 `OVERHAUL` \$1 `REPLACE` eventPayload.closureCode.resolvedModels + The set of models which called out the issue. + Type: List of Strings + Possible values: `vibrationISO` \$1 `vibrationML` \$1 `temperatureML` eventPayload.assetPositionId + The identifier of the asset position whose state changed. + Type: String models.temperatureML.persistentClassificationOutput + The persistent classification output from the machine learning based temperature model + Type: Number + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM` models.temperatureML.pointwiseClassificationOutput + The point–wise classification output from the machine learning based temperature model + Type: String + Valid Values: `UNKNOWN | INITIALIZING | HEALTHY | WARNING | ALARM` models.vibrationISO.isoClass + The ISO 20816 class (a standard for measurement and evaluation of machine vibration) used by the ISO based vibration model + Type: String + Valid Values: `CLASS1 | CLASS2 | CLASS3 | CLASS4` models.vibrationISO.mutedThreshold + The threshold to mute the notification from the ISO based vibration model + Type: String + Valid Values: `WARNING | ALARM` models.vibrationISO.persistentClassificationOutput + The persistent classification output from the ISO based vibration model + Type: String + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM` models.vibrationISO.pointwiseClassificationOutput + The point–wise classification output from the the ISO based vibration model + Type: String + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM | MUTED_WARNING | MUTED_ALARM` models.vibrationML.persistentClassificationOutput + The persistent classification output from the machine learning based vibration model + Type: String + Valid Values: `UNKNOWN | HEALTHY | WARNING | ALARM` models.vibrationML.pointwiseClassificationOutput + The point–wise classification output from the machine learning based vibration model + Type: String + Valid Values: `UNKNOWN | INITIALIZING | HEALTHY | WARNING | ALARM` assetState.newState + The machine status after processing the measurement + Type: String + Valid Values: `UNKNOWN | HEALTHY | NEEDS_MAINTENANCE | WARNING | ALARM` assetState.previousState + The machine status before processing the measurement + Type: String + Valid Values: `UNKNOWN | HEALTHY | NEEDS_MAINTENANCE | WARNING | ALARM` eventPayload.companyName + The name of the company using the asset. + Type: String eventPayload.geoLocation.latitude + The latitude of the site's physical location. + Type: Number eventPayload.geoLocation.longitude + The longitude of the site's physical location. + Type: Number eventPayload.address + The address of the site. + Type: String eventPayload.serialNumber + The serial number of the asset. + Type: String eventPayload.make + The make of the asset. + Type: String eventPayload.model + The model of the asset. + Type: String # Migration from Kinesis v1 to v2 If you are currently using the v1 data schema, you may already be sending data to Amazon S3, or further processing the data stream payload with Lambda. **Topics** + [Updating the data schema to v2](#updating-to-v2) + [Updating data processing with Lambda](#updating-with-lam) ## Updating the data schema to v2 If you have already configured a data stream with the v1 schema, you can update your data export process by doing the following: 1. Open your Amazon Monitron console. 1. Navigate to your project. 1. Stop the [current live data export](monitron-kinesis-export-v2.md#stop-kinesis-export-v2). 1. Start the live data export to create a new data stream. 1. Select the newly created data stream. 1. Choose **start live data export**. At this point, the new schema will send your payload through the data stream. 1. (Optional) Go to the Kinesis console and delete your old data stream. 1. Configure a new delivery method for your newly created data stream with the v2 schema. Your new stream now delivers payloads conforming to the v2 schema to your new bucket. We recommend using two distinct buckets to have a consistent format in case you want to process all the data in these buckets. For example, using other services such as Athena and AWS Glue. **Note** If you were delivering your data to Amazon S3, learn how to [store exported data in Amazon S3](kinesis-store-S3-v2.md#kinesis-store-S3-title-v2) for details on how to deliver your data to Amazon S3 with the v2 schema. **Note** If you were using a Lambda function to process your payloads, learn how to [ process data with Lambda](https://docs.aws.amazon.com/Monitron/latest/user-guide/data-export-lambda.html). You can also refer to the [updating with Lambda](#updating-with-lam) section for more information. ## Updating data processing with Lambda Updating the data processing with Lambda requires you to consider that the v2 data stream is now event-based. Your initial v1 Lambda code may have been similar to the following: ``` import base64 def main_handler(event): # Kinesis "data" blob is base64 encoded so decode here: for record in event['Records']: payload = base64.b64decode(record["kinesis"]["data"]) measurement = payload["measurement"] projectDisplayName = payload["projectDisplayName"] # Process the content of the measurement # ... ``` Since the v1 data schema is on a deprecation path, the previous Lambda code won't work with all the new data streams. The following Python sample code will process events from Kinesis stream with the data schema v2. This code uses the new `eventType` parameter to orient the processing to the appropriate handler: ``` import base64 handlers = { "measurement": measurementEventHandler, "gatewayConnected": gatewayConnectedEventHandler, "gatewayDisconnected": gatewayDisconnectedEventHandler, "sensorConnected": sensorConnectedEventHandler, "sensorDisconnected": sensorDisconnectedEventHandler, } def main_handler(event): # Kinesis "data" blob is base64 encoded so decode here: for record in event['Records']: payload = base64.b64decode(record["kinesis"]["data"]) eventType = payload["eventType"] if eventType not in handler.keys(): log.info("No event handler found for the event type: {event['eventType']}") return # Invoke the appropriate handler based on the event type. eventPayload = payload["eventPayload"] eventHandler = handlers[eventType] eventHandler(eventPayload) def measurementEventHandler(measurementEventPayload): # Handle measurement event projectName = measurementEventPayload["projectName"] # ... def gatewayConnectedEventHandler(gatewayConnectedEventPayload): # Handle gateway connected event # Other event handler functions ```