# Content delivery **Topics** + [Send AWS WAF logs to Splunk by using AWS Firewall Manager and Amazon Data Firehose](send-aws-waf-logs-to-splunk-by-using-aws-firewall-manager-and-amazon-data-firehose.md) + [Serve static content in an Amazon S3 bucket through a VPC by using Amazon CloudFront](serve-static-content-in-an-amazon-s3-bucket-through-a-vpc-by-using-amazon-cloudfront.md) + [More patterns](contentdelivery-more-patterns-pattern-list.md) # Send AWS WAF logs to Splunk by using AWS Firewall Manager and Amazon Data Firehose *Michael Friedenthal, Aman Kaur Gandhi, and JJ Johnson, Amazon Web Services* ## Summary Historically, there were two ways to move data into Splunk: a push or a pull architecture. A *pull architecture* offers delivery data guarantees through retries, but it requires dedicated resources in Splunk that poll data. Pull architectures usually are not real time because of the polling. A *push architecture* in typically has lower latency, is more scalable, and reduces operational complexity and costs. However, it doesn’t guarantee delivery and typically requires agents. Splunk integration with Amazon Data Firehose delivers real-time streaming data to Splunk through an HTTP event collector (HEC). This integration provides the advantages of both push and pull architectures—it guarantees data delivery through retries, is near real-time, and is low latency and low complexity. The HEC quickly and efficiently sends data over HTTP or HTTPS directly to Splunk. HECs are token-based, which eliminates the need to hardcode credentials in an application or in supporting files. In an AWS Firewall Manager policy, you can configure logging for all of the AWS WAF web ACL traffic in all of your accounts, and you can then use a Firehose delivery stream to send that log data to Splunk for monitoring, visualization, and analysis. This solution provides the following benefits: + Central management and logging for AWS WAF web ACL traffic in all of your accounts + Splunk integration with a single AWS account + Scalability + Near real-time delivery of log data + Cost optimization through the use of a serverless solution, so you don't have to pay for unused resources. ## Prerequisites and limitations **Prerequisites ** + An active AWS account that is part of an organization in AWS Organizations. + You must have the following permissions to enable logging with Firehose: + `iam:CreateServiceLinkedRole` + `firehose:ListDeliveryStreams` + `wafv2:PutLoggingConfiguration` + AWS WAF and its web ACLs must be configured. For instructions, see [Getting started with AWS WAF](https://docs.aws.amazon.com/waf/latest/developerguide/getting-started.html). + AWS Firewall Manager must be setup. For instructions, see [AWS Firewall Manager prerequisites](https://docs.aws.amazon.com/waf/latest/developerguide/fms-prereq.html). + The Firewall Manager security policies for AWS WAF must be configured. For instructions, see [Getting started with AWS Firewall ManagerAWS WAF policies](https://docs.aws.amazon.com/waf/latest/developerguide/getting-started-fms.html). + Splunk must be setup with a public HTTP endpoint that can be reached by Firehose. **Limitations** + The AWS accounts must be managed in a single organization in AWS Organizations. + The web ACL must be in the same Region as the delivery stream. If you are capturing logs for Amazon CloudFront, create the Firehose delivery stream in the US East (N. Virginia) Region, `us-east-1`. + The Splunk add-on for Firehose is available for paid Splunk Cloud deployments, distributed Splunk Enterprise deployments, and single-instance Splunk Enterprise deployments. This add-on is not supported for free trial Splunk Cloud deployments. ## Architecture **Target technology stack** + Firewall Manager + Firehose + Amazon Simple Storage Service (Amazon S3) + AWS WAF + Splunk **Target architecture ** The following image shows how you can use Firewall Manager to centrally log all AWS WAF data and send it to Splunk through Firehose. ![\[Architecture diagram showing sending AWS WAF log data to Splunk through Amazon Data Firehose\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/3dfeaae0-985a-42b8-91c4-ece081f0b51b/images/669169b1-caa4-419b-9988-19806ded54eb.png) 1. The AWS WAF web ACLs send firewall log data to Firewall Manager. 1. Firewall Manager sends the log data to Firehose. 1. The Firehose delivery stream forwards the log data to Splunk and to an S3 bucket. The S3 bucket acts as a backup in the event of an error with the Firehose delivery stream. **Automation and scale** This solution is designed to scale and accommodate all AWS WAF web ALCs within the organization. You can configure all web ACLs to use the same Firehose instance. However, if you want to set up and use multiple Firehose instances, you can. ## Tools **AWS services** + [AWS Firewall Manager](https://docs.aws.amazon.com/waf/latest/developerguide/fms-chapter.html) is a security management service that helps you to centrally configure and manage firewall rules across your accounts and applications in AWS Organizations. + [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) helps you deliver real-time [streaming data](http://aws.amazon.com/streaming-data/) to other AWS services, custom HTTP endpoints, and HTTP endpoints owned by supported third-party service providers, such as Splunk. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. + [AWS WAF](https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html) is a web application firewall that helps you monitor HTTP and HTTPS requests that are forwarded to your protected web application resources. **Other tools** + [Splunk](https://docs.splunk.com/Documentation) helps you monitor, visualize, and analyze log data. ## Epics ### Configure Splunk | Task | Description | Skills required | | --- | --- | --- | | Install the Splunk App for AWS. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-aws-waf-logs-to-splunk-by-using-aws-firewall-manager-and-amazon-data-firehose.html) | Security administrator, Splunk administrator | | Install the add-on for AWS WAF. | Repeat the previous instructions to install the **AWS Web Application Firewall Add-on** for Splunk. | Security administrator, Splunk administrator | | Install and configure the Splunk add-on for Firehose. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-aws-waf-logs-to-splunk-by-using-aws-firewall-manager-and-amazon-data-firehose.html) | Security administrator, Splunk administrator | ### Create the Firehose delivery stream | Task | Description | Skills required | | --- | --- | --- | | Grant Firehose access to a Splunk destination. | Configure the access policy that permits Firehose to access a Splunk destination and back up the log data to an S3 bucket. For more information, see [Grant Firehose access to a Splunk destination](https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#using-iam-splunk). | Security administrator | | Create a Firehose delivery stream. | In the same account where you manage the web ACLs for AWS WAF, create a delivery stream in Firehose. You are required to have an IAM role when creating a delivery stream. Firehose assumes that IAM role and gains access to the specified S3 bucket. For instructions, see [Creating a delivery stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html). Note the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/send-aws-waf-logs-to-splunk-by-using-aws-firewall-manager-and-amazon-data-firehose.html)Repeat this process for each token that you configured in the HTTP event collector. | Security administrator | | Test the delivery stream. | Test the delivery stream to validate that it is properly configured. For instructions, see [Test using Splunk as the destination](https://docs.aws.amazon.com/firehose/latest/dev/test-drive-firehose.html#test-drive-destination-splunk) in the Firehose documentation. | Security administrator | ### Configure Firewall Manager to log data | Task | Description | Skills required | | --- | --- | --- | | Configure the Firewall Manager policies. | The Firewall Manager policies must be configured to enable logging and to forward logs to the correct Firehose delivery stream. For more information and instructions, see [Configuring logging for an AWS WAF policy](https://docs.aws.amazon.com/waf/latest/developerguide/waf-policies.html#waf-policies-logging-config). | Security administrator | ## Related resources **AWS resources** + [Logging web ACL traffic](https://docs.aws.amazon.com/waf/latest/developerguide/logging.html) (AWS WAF documentation) + [Configuring logging for an AWS WAF policy](https://docs.aws.amazon.com/waf/latest/developerguide/waf-policies.html#waf-policies-logging-config) (AWS WAF documentation) + [Tutorial: Sending VPC Flow Logs to Splunk Using Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/vpc-splunk-tutorial.html) (Firehose documentation) + [How do I push VPC flow logs to Splunk using Amazon Data Firehose?](https://aws.amazon.com/premiumsupport/knowledge-center/push-flow-logs-splunk-firehose/) (AWS Knowledge Center) + [Power data ingestion into Splunk using Amazon Data Firehose](https://aws.amazon.com/blogs/big-data/power-data-ingestion-into-splunk-using-amazon-kinesis-data-firehose/) (AWS blog post) **Splunk documentation** + [Splunk Add-on for Amazon Data Firehose](https://docs.splunk.com/Documentation/AddOns/released/Firehose/About) # Serve static content in an Amazon S3 bucket through a VPC by using Amazon CloudFront *Angel Emmanuel Hernandez Cebrian, Amazon Web Services* ## Summary When you serve static content that is hosted on Amazon Web Services (AWS), the recommended approach is to use an Amazon Simple Storage Service (S3) bucket as the origin and use Amazon CloudFront to distribute the content. This solution has two primary benefits: the convenience of caching static content at edge locations, and the ability to define [web access control lists](https://docs.aws.amazon.com/waf/latest/developerguide/web-acl.html) (web ACLs) for the CloudFront distribution, which helps you secure requests to the content with minimal configuration and administrative overhead. However, there is a common architectural limitation to the standard, recommended approach. In some environments, you want virtual firewall appliances deployed in a virtual private cloud (VPC) to inspect all content, including static content. The standard approach doesn’t route traffic through the VPC for inspection. This pattern provides an alternative architectural solution. You still use a CloudFront distribution to serve static content in an S3 bucket, but the traffic is routed through the VPC by using an Application Load Balancer. An AWS Lambda function then retrieves and returns the content from the S3 bucket. ## Prerequisites and limitations **Prerequisites** + An active AWS account. + Static website content hosted in an S3 bucket. **Limitations** + The resources in this pattern must be in a single AWS Region, but they can be provisioned in different AWS accounts. + Limits apply to the maximum request and response size that the Lambda function can receive and send, respectively. For more information, see *Limits* in [Lambda functions as targets](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html) (Elastic Load Balancing documentation). + It's important to find a good balance between performance, scalability, security, and cost-effectiveness when using this approach. Despite the high scalability of Lambda, if the number of concurrent Lambda invocations exceeds the maximum quota, some requests are throttled. For more information, see Lambda quotas (Lambda documentation). You also need to consider pricing when using Lambda. To minimize Lambda invocations, make sure that you properly define the cache for the CloudFront distribution. For more information, see [Optimizing caching and availability](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ConfiguringCaching.html) (CloudFront documentation). ## Architecture **Target technology stack  ** + CloudFront + Amazon Virtual Private Cloud (Amazon VPC) + Application Load Balancer + Lambda + Amazon S3 **Target architecture** The following image shows the suggested architecture when you need to use CloudFront to serve static content from an S3 bucket through a VPC. ![\[Traffic flow through Application Load Balancers in the VPC to the Lambda function.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/e0dd6928-4fe0-47ab-954f-9de5563349d8/images/b42c7dd9-4a72-4998-bf88-195c8f90ed3e.png) 1. The client requests the URL of CloudFront distribution to get a particular website file in the S3 bucket. 1. CloudFront sends the request to AWS WAF. AWS WAF filters the request by using the web ACLs applied to the CloudFront distribution. If the request is determined to be valid, the flow continues. If the request is determined to be invalid, the client receives a 403 error. 1. CloudFront checks its internal cache. If there is a valid key matching the incoming request, the associated value is sent back to the client as a response. If not, the flow continues. 1. CloudFront forwards the request to the URL of the specified Application Load Balancer. 1. The Application Load Balancer has a listener associated with a target group based on a Lambda function. The Application Load Balancer invokes the Lambda function. 1. The Lambda function connects to the S3 bucket, perform a `GetObject` operation on it, and returns the content as a response. **Automation and scale** To automate the deployment of static content using this approach, create CI/CD pipelines to update the Amazon S3 buckets that host websites. The Lambda function scales automatically to handle the concurrent requests, within the quotas and limitations of the service. For more information, see [Lambda function scaling](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html) and [Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) (Lambda documentation). For the other AWS services and features, such as CloudFront and the Application Load Balancer, AWS scales these automatically. ## Tools + [Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Introduction.html) speeds up distribution of your web content by delivering it through a worldwide network of data centers, which lowers latency and improves performance. + [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html) distributes incoming application or network traffic across multiple targets. In this pattern, you use an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html) provisioned through Elastic Load Balancing to direct traffic to the Lambda function. + [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use. + [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data. + [Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS. ## Epics ### Use CloudFront to serve static content from Amazon S3 through a VPC | Task | Description | Skills required | | --- | --- | --- | | Create a VPC. | Create a VPC for hosting the resources deployed in this pattern, such as the Application Load Balancer and the Lambda function.  For instructions, see [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-vpcs.html#Create-VPC) (Amazon VPC documentation). | Cloud architect | | Create an AWS WAF web ACL. | Create an AWS WAF web ACL. Later in this pattern, you apply this web ACL to the CloudFront distribution. For instructions, see [Creating a web ACL](https://docs.aws.amazon.com/waf/latest/developerguide/web-acl-creating.html) (AWS WAF documentation). | Cloud architect | | Create the Lambda function. | Create the Lambda function that serves the static content hosted in the S3 bucket as a website. Use the code provided in the [Additional information](#serve-static-content-in-an-amazon-s3-bucket-through-a-vpc-by-using-amazon-cloudfront-additional) section of this pattern. Customize the code to identify your target S3 bucket. | General AWS | | Upload the Lambda function. | Enter the following command to upload the Lambda function code to a .zip file archive in Lambda.
aws lambda update-function-code \
--function-name  \ 
--zip-file fileb://lambda-alb-s3-website.zip
| General AWS | | Create an Application Load Balancer. | Create an internet-facing Application Load Balancer that points to the Lambda function. For instructions, see [Create a target group for the Lambda function](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#register-lambda-function) (Elastic Load Balancing documentation). For a high-availability configuration, create the Application Load Balancer and attach it to private subnets in different Availability Zones. | Cloud architect | | Create a CloudFront distribution. | Create a CloudFront distribution that points to the Application Load Balancer you created.[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/serve-static-content-in-an-amazon-s3-bucket-through-a-vpc-by-using-amazon-cloudfront.html) | Cloud architect | ## Related resources **AWS documentation** + [Optimizing caching and availability](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ConfiguringCaching.html) (CloudFront documentation) + [Lambda functions as targets](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html) (Elastic Load Balancing documentation) + [Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) (Lambda documentation) **AWS service websites** + [Application Load Balancer](https://aws.amazon.com/es/elasticloadbalancing/application-load-balancer/) + [Lambda](https://aws.amazon.com/en/lambda/) + [CloudFront](https://aws.amazon.com/en/cloudfront/) + [Amazon S3](https://aws.amazon.com/en/s3/) + [AWS WAF](https://aws.amazon.com/en/waf/) + [Amazon VPC](https://aws.amazon.com/en/vpc/) ## Additional information **Code** The following example Lambda function is written in Node.js. This Lambda function acts as a web server that performs a `GetObject` operation to an S3 bucket that contains the website resources. ``` /** * This is an AWS Lambda function created for demonstration purposes. * It retrieves static assets from a defined Amazon S3 bucket. * To make the content available through a URL, use an Application Load Balancer with a Lambda integration. * * Set the S3_BUCKET environment variable in the Lambda function definition. */ var AWS = require('aws-sdk'); exports.handler = function(event, context, callback) { var bucket = process.env.S3_BUCKET; var key = event.path.replace('/', ''); if (key == '') { key = 'index.html'; } // Fetch from S3 var s3 = new AWS.S3(); return s3.getObject({Bucket: bucket, Key: key}, function(err, data) { if (err) { return err; } var isBase64Encoded = false; var encoding = 'utf8'; if (data.ContentType.indexOf('image/') > -1) { isBase64Encoded = true; encoding = 'base64' } var resp = { statusCode: 200, headers: { 'Content-Type': data.ContentType, }, body: new Buffer(data.Body).toString(encoding), isBase64Encoded: isBase64Encoded }; callback(null, resp); } ); }; ``` # More patterns **Topics** + [Check an Amazon CloudFront distribution for access logging, HTTPS, and TLS version](check-an-amazon-cloudfront-distribution-for-access-logging-https-and-tls-version.md) + [Deploy a gRPC-based application on an Amazon EKS cluster and access it with an Application Load Balancer](deploy-a-grpc-based-application-on-an-amazon-eks-cluster-and-access-it-with-an-application-load-balancer.md) + [Deploy preventative attribute-based access controls for public subnets](deploy-preventative-attribute-based-access-controls-for-public-subnets.md) + [Deploy resources in an AWS Wavelength Zone by using Terraform](deploy-resources-wavelength-zone-using-terraform.md) + [Deploy the Security Automations for AWS WAF solution by using Terraform](deploy-the-security-automations-for-aws-waf-solution-by-using-terraform.md) + [Set up a serverless cell router for a cell-based architecture](serverless-cell-router-architecture.md) + [Use Amazon Q Developer as a coding assistant to increase your productivity](use-q-developer-as-coding-assistant-to-increase-productivity.md) + [View AWS Network Firewall logs and metrics by using Splunk](view-aws-network-firewall-logs-and-metrics-by-using-splunk.md)