Enabling Security Lake programmatically

This tutorial explains how to enable and start using Security Lake programmatically. The Amazon Security Lake API gives you comprehensive, programmatic access to your Security Lake account, data, and resources. Alternatively, you can use AWS command line tools— the AWS Command Line Interface or the AWS Tools for PowerShell—or the AWS SDKs to access Security Lake.

Step 1: Create IAM roles

If you access Security Lake programmatically, it's necessary to create some AWS Identity and Access Management (IAM) roles in order to configure your data lake.

You must create roles in IAM if you'll be taking one or more of the following actions (choose the links to see more information about IAM roles for each action):

After creating the roles previously mentioned, attach the AmazonSecurityLakeAdministrator AWS managed policy to the role that you're using to enable Security Lake. This policy grants administrative permissions that allow a principal to onboard to Security Lake and access all Security Lake actions.

Attach the AmazonSecurityLakeMetaStoreManager AWS managed policy to create your data lake or query data from Security Lake. This policy is necessary for Security Lake to support extract, transform, and load (ETL) jobs on raw log and event data that it receives from sources.

Step 2: Enable Amazon Security Lake

To enable Security Lake programmatically, use the CreateDataLake operation of the Security Lake API. If you're using the AWS CLI, run the create-data-lake command. In your request, use the region field of the configurations object to specify the Region code for the Region in which to enable Security Lake. For a list of Region codes, see Amazon Security Lake endpoints in the AWS General Reference.

Example 1

The following example command enables Security Lake in the us-east-1 and us-east-2 Regions. In both Regions, this data lake is encrypted with Amazon S3 managed keys. Objects expire after 365 days, and objects transition to the ONEZONE_IA S3 storage class after 60 days. This example is formatted for Linux, macOS, or Unix, and it uses the backslash (\) line-continuation character to improve readability.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-1","lifecycleConfiguration": {"expiration":{"days":365},"transitions":[{"days":60,"storageClass":"ONEZONE_IA"}]}}, {"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-2","lifecycleConfiguration": {"expiration":{"days":365},"transitions":[{"days":60,"storageClass":"ONEZONE_IA"}]}}]' \
--meta-store-manager-role-arn "arn:aws:iam:us-east-1:123456789012:role/service-role/AmazonSecurityLakeMetaStoreManager"

Example 2

The following example command enables Security Lake in the us-east-2 Region. This data lake is encrypted with a customer managed key that was created in AWS Key Management Service (AWS KMS). Objects expire after 500 days, and objects transition to the GLACIER S3 storage class after 30 days. This example is formatted for Linux, macOS, or Unix, and it uses the backslash (\) line-continuation character to improve readability.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"1234abcd-12ab-34cd-56ef-1234567890ab"},"region":"us-east-2","lifecycleConfiguration": {"expiration":{"days":500},"transitions":[{"days":30,"storageClass":"GLACIER"}]}}]' \
--meta-store-manager-role-arn "arn:aws:iam:us-east-1:123456789012:role/service-role/AmazonSecurityLakeMetaStoreManager"

Step 3: Configure sources

Security Lake collects log and event data from a variety of sources and across your AWS accounts and AWS Regions. Follow these instructions to identify which data you want Security Lake to collect. You can only use these instructions to add a natively-supported AWS service as a source. For information about adding a custom source, see Collecting data from custom sources in Security Lake.

To define one or more collection sources programmatically, use the CreateAwsLogSource operation of the Security Lake API. For each source, specify a Regionally unique value for the sourceName parameter. Optionally use additional parameters to limit the scope of the source to specific accounts (accounts) or a specific version (sourceVersion).

If your request specifies a Region in which you haven't enabled Security Lake, an error occurs. To address this error, ensure that the regions array specifies only those Regions in which you've enabled Security Lake. Alternatively, you can enable Security Lake in the Region, and then submit your request again.

When you enable Security Lake in an account for the first time, all the selected log and event sources will be a part of a 15-day free trial period. For more information about usage statistics, see Reviewing usage and estimated costs.

Step 4: Configure storage settings and rollup Regions (optional)

You can specify the Amazon S3 storage class in which you want Security Lake to store your data and for how long. You can also specify a rollup Region to consolidate data from multiple Regions. These are optional steps. For more information, see Lifecycle management in Security Lake.

To define a target objective programmatically when you enable Security Lake, use the CreateDataLake operation of the Security Lake API. If you've already enabled Security Lake and want to define a target objective, use the UpdateDataLake operation, not the CreateDataLake operation.

For either operation, use the supported parameters to specify the configuration settings that you want:

For example, the following command creates a data lake with ap-northeast-2 as a rollup Region. The us-east-1 Region will contribute data to the ap-northeast-2 Region. This example also establishes a 10-day expiration period for objects that are added to the data lake.

$ aws securitylake create-data-lake \
--configurations '[{"encryptionConfiguration": {"kmsKeyId":"S3_MANAGED_KEY"},"region":"us-east-1","replicationConfiguration": {"regions": ["ap-northeast-2"],"roleArn":"arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeS3ReplicationRole"},"lifecycleConfiguration": {"expiration":{"days":10}}}]' \
--meta-store-manager-role-arn "arn:aws:iam::123456789012:role/service-role/AmazonSecurityLakeMetaStoreManager"

You have now created your data lake. Use the ListDataLakes operation of the Security Lake API to verify enablement of Security Lake and your data lake settings in each Region.

If issues or errors arise in the creation of your data lake, you can view a list of exceptions by using the ListDataLakeExceptions operation, and notify users of exceptions with the CreateDataLakeExceptionSubscription operation. For more information, see Troubleshooting data lake status.

Step 5: View and query your own data

After creating your data lake, you can use Amazon Athena or similar services to view and query your data from AWS Lake Formation databases and tables. When you programmatically enable Security Lake, database view permissions aren't granted automatically. The data lake administrator account in AWS Lake Formation must grant SELECT permissions to the IAM role you want to use to query the relevant databases and tables. At a minimum, the role must have Data analyst permissions. For more information on permission levels, see Lake Formation personas and IAM permissions reference. For instructions on granting SELECT permissions, see Granting Data Catalog permissions using the named resource method in the AWS Lake Formation Developer Guide.

Step 6: Create subscribers

After creating your data lake, you can add subscribers to consume your data. Subscribers can consume data by directly accessing objects in your Amazon S3 buckets or by querying the data lake. For more information about subscribers, see Subscriber management in Security Lake.