# Amazon S3 examples using AWS CLI with Bash script
The following code examples show you how to perform actions and implement common scenarios by using the AWS Command Line Interface with Bash script with Amazon S3.
*Basics* are code examples that show you how to perform the essential operations within a service.
*Actions* are code excerpts from larger programs and must be run in context. While actions show you how to call individual service functions, you can see actions in context in their related scenarios.
*Scenarios* are code examples that show you how to accomplish specific tasks by calling multiple functions within a service or combined with other AWS services.
Each example includes a link to the complete source code, where you can find instructions on how to set up and run the code in context.
**Topics**
+ [Basics](#basics)
+ [Actions](#actions)
+ [Scenarios](#scenarios)
## Basics
### Learn the basics
The following code example shows how to:
+ Create a bucket and upload a file to it.
+ Download an object from a bucket.
+ Copy an object to a subfolder in a bucket.
+ List the objects in a bucket.
+ Delete the bucket objects and the bucket.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function s3_getting_started
#
# This function creates, copies, and deletes S3 buckets and objects.
#
# Returns:
# 0 - If successful.
# 1 - If an error occurred.
###############################################################################
function s3_getting_started() {
{
if [ "$BUCKET_OPERATIONS_SOURCED" != "True" ]; then
cd bucket-lifecycle-operations || exit
source ./bucket_operations.sh
cd ..
fi
}
echo_repeat "*" 88
echo "Welcome to the Amazon S3 getting started demo."
echo_repeat "*" 88
echo "A unique bucket will be created by appending a Universally Unique Identifier to a bucket name prefix."
echo -n "Enter a prefix for the S3 bucket that will be used in this demo: "
get_input
bucket_name_prefix=$get_input_result
local bucket_name
bucket_name=$(generate_random_name "$bucket_name_prefix")
local region_code
region_code=$(aws configure get region)
if create_bucket -b "$bucket_name" -r "$region_code"; then
echo "Created demo bucket named $bucket_name"
else
errecho "The bucket failed to create. This demo will exit."
return 1
fi
local file_name
while [ -z "$file_name" ]; do
echo -n "Enter a file you want to upload to your bucket: "
get_input
file_name=$get_input_result
if [ ! -f "$file_name" ]; then
echo "Could not find file $file_name. Are you sure it exists?"
file_name=""
fi
done
local key
key="$(basename "$file_name")"
local result=0
if copy_file_to_bucket "$bucket_name" "$file_name" "$key"; then
echo "Uploaded file $file_name into bucket $bucket_name with key $key."
else
result=1
fi
local destination_file
destination_file="$file_name.download"
if yes_no_input "Would you like to download $key to the file $destination_file? (y/n) "; then
if download_object_from_bucket "$bucket_name" "$destination_file" "$key"; then
echo "Downloaded $key in the bucket $bucket_name to the file $destination_file."
else
result=1
fi
fi
if yes_no_input "Would you like to copy $key a new object key in your bucket? (y/n) "; then
local to_key
to_key="demo/$key"
if copy_item_in_bucket "$bucket_name" "$key" "$to_key"; then
echo "Copied $key in the bucket $bucket_name to the $to_key."
else
result=1
fi
fi
local bucket_items
bucket_items=$(list_items_in_bucket "$bucket_name")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
result=1
fi
echo "Your bucket contains the following items."
echo -e "Name\t\tSize"
echo "$bucket_items"
if yes_no_input "Delete the bucket, $bucket_name, as well as the objects in it? (y/n) "; then
bucket_items=$(echo "$bucket_items" | cut -f 1)
if delete_items_in_bucket "$bucket_name" "$bucket_items"; then
echo "The following items were deleted from the bucket $bucket_name"
echo "$bucket_items"
else
result=1
fi
if delete_bucket "$bucket_name"; then
echo "Deleted the bucket $bucket_name"
else
result=1
fi
fi
return $result
}
```
The Amazon S3 functions used in this scenario.
```
###############################################################################
# function create-bucket
#
# This function creates the specified bucket in the specified AWS Region, unless
# it already exists.
#
# Parameters:
# -b bucket_name -- The name of the bucket to create.
# -r region_code -- The code for an AWS Region in which to
# create the bucket.
#
# Returns:
# The URL of the bucket that was created.
# And:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function create_bucket() {
local bucket_name region_code response
local option OPTARG # Required to use getopts command in a function.
# bashsupport disable=BP5008
function usage() {
echo "function create_bucket"
echo "Creates an Amazon S3 bucket. You must supply a bucket name:"
echo " -b bucket_name The name of the bucket. It must be globally unique."
echo " [-r region_code] The code for an AWS Region in which the bucket is created."
echo ""
}
# Retrieve the calling parameters.
while getopts "b:r:h" option; do
case "${option}" in
b) bucket_name="${OPTARG}" ;;
r) region_code="${OPTARG}" ;;
h)
usage
return 0
;;
\?)
echo "Invalid parameter"
usage
return 1
;;
esac
done
if [[ -z "$bucket_name" ]]; then
errecho "ERROR: You must provide a bucket name with the -b parameter."
usage
return 1
fi
local bucket_config_arg
# A location constraint for "us-east-1" returns an error.
if [[ -n "$region_code" ]] && [[ "$region_code" != "us-east-1" ]]; then
bucket_config_arg="--create-bucket-configuration LocationConstraint=$region_code"
fi
iecho "Parameters:\n"
iecho " Bucket name: $bucket_name"
iecho " Region code: $region_code"
iecho ""
# If the bucket already exists, we don't want to try to create it.
if (bucket_exists "$bucket_name"); then
errecho "ERROR: A bucket with that name already exists. Try again."
return 1
fi
# shellcheck disable=SC2086
response=$(aws s3api create-bucket \
--bucket "$bucket_name" \
$bucket_config_arg)
# shellcheck disable=SC2181
if [[ ${?} -ne 0 ]]; then
errecho "ERROR: AWS reports create-bucket operation failed.\n$response"
return 1
fi
}
###############################################################################
# function copy_file_to_bucket
#
# This function creates a file in the specified bucket.
#
# Parameters:
# $1 - The name of the bucket to copy the file to.
# $2 - The path and file name of the local file to copy to the bucket.
# $3 - The key (name) to call the copy of the file in the bucket.
#
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function copy_file_to_bucket() {
local response bucket_name source_file destination_file_name
bucket_name=$1
source_file=$2
destination_file_name=$3
response=$(aws s3api put-object \
--bucket "$bucket_name" \
--body "$source_file" \
--key "$destination_file_name")
# shellcheck disable=SC2181
if [[ ${?} -ne 0 ]]; then
errecho "ERROR: AWS reports put-object operation failed.\n$response"
return 1
fi
}
###############################################################################
# function download_object_from_bucket
#
# This function downloads an object in a bucket to a file.
#
# Parameters:
# $1 - The name of the bucket to download the object from.
# $2 - The path and file name to store the downloaded bucket.
# $3 - The key (name) of the object in the bucket.
#
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function download_object_from_bucket() {
local bucket_name=$1
local destination_file_name=$2
local object_name=$3
local response
response=$(aws s3api get-object \
--bucket "$bucket_name" \
--key "$object_name" \
"$destination_file_name")
# shellcheck disable=SC2181
if [[ ${?} -ne 0 ]]; then
errecho "ERROR: AWS reports put-object operation failed.\n$response"
return 1
fi
}
###############################################################################
# function copy_item_in_bucket
#
# This function creates a copy of the specified file in the same bucket.
#
# Parameters:
# $1 - The name of the bucket to copy the file from and to.
# $2 - The key of the source file to copy.
# $3 - The key of the destination file.
#
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function copy_item_in_bucket() {
local bucket_name=$1
local source_key=$2
local destination_key=$3
local response
response=$(aws s3api copy-object \
--bucket "$bucket_name" \
--copy-source "$bucket_name/$source_key" \
--key "$destination_key")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api copy-object operation failed.\n$response"
return 1
fi
}
###############################################################################
# function list_items_in_bucket
#
# This function displays a list of the files in the bucket with each file's
# size. The function uses the --query parameter to retrieve only the key and
# size fields from the Contents collection.
#
# Parameters:
# $1 - The name of the bucket.
#
# Returns:
# The list of files in text format.
# And:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function list_items_in_bucket() {
local bucket_name=$1
local response
response=$(aws s3api list-objects \
--bucket "$bucket_name" \
--output text \
--query 'Contents[].{Key: Key, Size: Size}')
# shellcheck disable=SC2181
if [[ ${?} -eq 0 ]]; then
echo "$response"
else
errecho "ERROR: AWS reports s3api list-objects operation failed.\n$response"
return 1
fi
}
###############################################################################
# function delete_items_in_bucket
#
# This function deletes the specified list of keys from the specified bucket.
#
# Parameters:
# $1 - The name of the bucket.
# $2 - A list of keys in the bucket to delete.
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function delete_items_in_bucket() {
local bucket_name=$1
local keys=$2
local response
# Create the JSON for the items to delete.
local delete_items
delete_items="{\"Objects\":["
for key in $keys; do
delete_items="$delete_items{\"Key\": \"$key\"},"
done
delete_items=${delete_items%?} # Remove the final comma.
delete_items="$delete_items]}"
response=$(aws s3api delete-objects \
--bucket "$bucket_name" \
--delete "$delete_items")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api delete-object operation failed.\n$response"
return 1
fi
}
###############################################################################
# function delete_bucket
#
# This function deletes the specified bucket.
#
# Parameters:
# $1 - The name of the bucket.
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function delete_bucket() {
local bucket_name=$1
local response
response=$(aws s3api delete-bucket \
--bucket "$bucket_name")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api delete-bucket failed.\n$response"
return 1
fi
}
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [CopyObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CopyObject)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteObjects](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteObjects)
+ [GetObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/GetObject)
+ [ListObjectsV2](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/ListObjectsV2)
+ [PutObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutObject)
## Actions
### `CopyObject`
The following code example shows how to use `CopyObject`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function copy_item_in_bucket
#
# This function creates a copy of the specified file in the same bucket.
#
# Parameters:
# $1 - The name of the bucket to copy the file from and to.
# $2 - The key of the source file to copy.
# $3 - The key of the destination file.
#
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function copy_item_in_bucket() {
local bucket_name=$1
local source_key=$2
local destination_key=$3
local response
response=$(aws s3api copy-object \
--bucket "$bucket_name" \
--copy-source "$bucket_name/$source_key" \
--key "$destination_key")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api copy-object operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [CopyObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CopyObject) in *AWS CLI Command Reference*.
### `CreateBucket`
The following code example shows how to use `CreateBucket`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function iecho
#
# This function enables the script to display the specified text only if
# the global variable $VERBOSE is set to true.
###############################################################################
function iecho() {
if [[ $VERBOSE == true ]]; then
echo "$@"
fi
}
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function create-bucket
#
# This function creates the specified bucket in the specified AWS Region, unless
# it already exists.
#
# Parameters:
# -b bucket_name -- The name of the bucket to create.
# -r region_code -- The code for an AWS Region in which to
# create the bucket.
#
# Returns:
# The URL of the bucket that was created.
# And:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function create_bucket() {
local bucket_name region_code response
local option OPTARG # Required to use getopts command in a function.
# bashsupport disable=BP5008
function usage() {
echo "function create_bucket"
echo "Creates an Amazon S3 bucket. You must supply a bucket name:"
echo " -b bucket_name The name of the bucket. It must be globally unique."
echo " [-r region_code] The code for an AWS Region in which the bucket is created."
echo ""
}
# Retrieve the calling parameters.
while getopts "b:r:h" option; do
case "${option}" in
b) bucket_name="${OPTARG}" ;;
r) region_code="${OPTARG}" ;;
h)
usage
return 0
;;
\?)
echo "Invalid parameter"
usage
return 1
;;
esac
done
if [[ -z "$bucket_name" ]]; then
errecho "ERROR: You must provide a bucket name with the -b parameter."
usage
return 1
fi
local bucket_config_arg
# A location constraint for "us-east-1" returns an error.
if [[ -n "$region_code" ]] && [[ "$region_code" != "us-east-1" ]]; then
bucket_config_arg="--create-bucket-configuration LocationConstraint=$region_code"
fi
iecho "Parameters:\n"
iecho " Bucket name: $bucket_name"
iecho " Region code: $region_code"
iecho ""
# If the bucket already exists, we don't want to try to create it.
if (bucket_exists "$bucket_name"); then
errecho "ERROR: A bucket with that name already exists. Try again."
return 1
fi
# shellcheck disable=SC2086
response=$(aws s3api create-bucket \
--bucket "$bucket_name" \
$bucket_config_arg)
# shellcheck disable=SC2181
if [[ ${?} -ne 0 ]]; then
errecho "ERROR: AWS reports create-bucket operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket) in *AWS CLI Command Reference*.
### `DeleteBucket`
The following code example shows how to use `DeleteBucket`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function delete_bucket
#
# This function deletes the specified bucket.
#
# Parameters:
# $1 - The name of the bucket.
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function delete_bucket() {
local bucket_name=$1
local response
response=$(aws s3api delete-bucket \
--bucket "$bucket_name")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api delete-bucket failed.\n$response"
return 1
fi
}
```
+ For API details, see [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket) in *AWS CLI Command Reference*.
### `DeleteObject`
The following code example shows how to use `DeleteObject`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function delete_item_in_bucket
#
# This function deletes the specified file from the specified bucket.
#
# Parameters:
# $1 - The name of the bucket.
# $2 - The key (file name) in the bucket to delete.
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function delete_item_in_bucket() {
local bucket_name=$1
local key=$2
local response
response=$(aws s3api delete-object \
--bucket "$bucket_name" \
--key "$key")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api delete-object operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [DeleteObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteObject) in *AWS CLI Command Reference*.
### `DeleteObjects`
The following code example shows how to use `DeleteObjects`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function delete_items_in_bucket
#
# This function deletes the specified list of keys from the specified bucket.
#
# Parameters:
# $1 - The name of the bucket.
# $2 - A list of keys in the bucket to delete.
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function delete_items_in_bucket() {
local bucket_name=$1
local keys=$2
local response
# Create the JSON for the items to delete.
local delete_items
delete_items="{\"Objects\":["
for key in $keys; do
delete_items="$delete_items{\"Key\": \"$key\"},"
done
delete_items=${delete_items%?} # Remove the final comma.
delete_items="$delete_items]}"
response=$(aws s3api delete-objects \
--bucket "$bucket_name" \
--delete "$delete_items")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]]; then
errecho "ERROR: AWS reports s3api delete-object operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [DeleteObjects](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteObjects) in *AWS CLI Command Reference*.
### `GetObject`
The following code example shows how to use `GetObject`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function download_object_from_bucket
#
# This function downloads an object in a bucket to a file.
#
# Parameters:
# $1 - The name of the bucket to download the object from.
# $2 - The path and file name to store the downloaded bucket.
# $3 - The key (name) of the object in the bucket.
#
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function download_object_from_bucket() {
local bucket_name=$1
local destination_file_name=$2
local object_name=$3
local response
response=$(aws s3api get-object \
--bucket "$bucket_name" \
--key "$object_name" \
"$destination_file_name")
# shellcheck disable=SC2181
if [[ ${?} -ne 0 ]]; then
errecho "ERROR: AWS reports put-object operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [GetObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/GetObject) in *AWS CLI Command Reference*.
### `HeadBucket`
The following code example shows how to use `HeadBucket`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function bucket_exists
#
# This function checks to see if the specified bucket already exists.
#
# Parameters:
# $1 - The name of the bucket to check.
#
# Returns:
# 0 - If the bucket already exists.
# 1 - If the bucket doesn't exist.
###############################################################################
function bucket_exists() {
local bucket_name
bucket_name=$1
# Check whether the bucket already exists.
# We suppress all output - we're interested only in the return code.
if aws s3api head-bucket \
--bucket "$bucket_name" \
>/dev/null 2>&1; then
return 0 # 0 in Bash script means true.
else
return 1 # 1 in Bash script means false.
fi
}
```
+ For API details, see [HeadBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/HeadBucket) in *AWS CLI Command Reference*.
### `ListObjectsV2`
The following code example shows how to use `ListObjectsV2`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function list_items_in_bucket
#
# This function displays a list of the files in the bucket with each file's
# size. The function uses the --query parameter to retrieve only the key and
# size fields from the Contents collection.
#
# Parameters:
# $1 - The name of the bucket.
#
# Returns:
# The list of files in text format.
# And:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function list_items_in_bucket() {
local bucket_name=$1
local response
response=$(aws s3api list-objects \
--bucket "$bucket_name" \
--output text \
--query 'Contents[].{Key: Key, Size: Size}')
# shellcheck disable=SC2181
if [[ ${?} -eq 0 ]]; then
echo "$response"
else
errecho "ERROR: AWS reports s3api list-objects operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [ListObjectsV2](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/ListObjectsV2) in *AWS CLI Command Reference*.
### `PutObject`
The following code example shows how to use `PutObject`.
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/aws-cli/bash-linux/s3#code-examples).
```
###############################################################################
# function errecho
#
# This function outputs everything sent to it to STDERR (standard error output).
###############################################################################
function errecho() {
printf "%s\n" "$*" 1>&2
}
###############################################################################
# function copy_file_to_bucket
#
# This function creates a file in the specified bucket.
#
# Parameters:
# $1 - The name of the bucket to copy the file to.
# $2 - The path and file name of the local file to copy to the bucket.
# $3 - The key (name) to call the copy of the file in the bucket.
#
# Returns:
# 0 - If successful.
# 1 - If it fails.
###############################################################################
function copy_file_to_bucket() {
local response bucket_name source_file destination_file_name
bucket_name=$1
source_file=$2
destination_file_name=$3
response=$(aws s3api put-object \
--bucket "$bucket_name" \
--body "$source_file" \
--key "$destination_file_name")
# shellcheck disable=SC2181
if [[ ${?} -ne 0 ]]; then
errecho "ERROR: AWS reports put-object operation failed.\n$response"
return 1
fi
}
```
+ For API details, see [PutObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutObject) in *AWS CLI Command Reference*.
## Scenarios
### Getting started with Amazon Athena
The following code example shows how to:
+ Create an S3 bucket for query results
+ Create a database
+ Create a table
+ Run a query
+ Create and use named queries
+ Clean up resources
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/061-amazon-athena-gs) repository.
```
#!/bin/bash
# Amazon Athena Getting Started Script
# This script demonstrates how to use Amazon Athena with AWS CLI
# It creates a database, table, runs queries, and manages named queries
# Set up logging
LOG_FILE="athena-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting Amazon Athena Getting Started Tutorial..."
echo "Logging to $LOG_FILE"
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Resources created:"
if [ -n "$NAMED_QUERY_ID" ]; then
echo "- Named Query: $NAMED_QUERY_ID"
fi
if [ -n "$DATABASE_NAME" ]; then
echo "- Database: $DATABASE_NAME"
if [ -n "$TABLE_NAME" ]; then
echo "- Table: $TABLE_NAME in $DATABASE_NAME"
fi
fi
if [ -n "$S3_BUCKET" ]; then
echo "- S3 Bucket: $S3_BUCKET"
fi
echo "Exiting..."
exit 1
}
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
S3_BUCKET="athena-${RANDOM_ID}"
DATABASE_NAME="mydatabase"
TABLE_NAME="cloudfront_logs"
# Get the current AWS region
AWS_REGION=$(aws configure get region)
if [ -z "$AWS_REGION" ]; then
AWS_REGION="us-east-1"
echo "No AWS region found in configuration, defaulting to $AWS_REGION"
fi
echo "Using AWS Region: $AWS_REGION"
# Create S3 bucket for Athena query results
echo "Creating S3 bucket for Athena query results: $S3_BUCKET"
CREATE_BUCKET_RESULT=$(aws s3 mb "s3://$S3_BUCKET" 2>&1)
if echo "$CREATE_BUCKET_RESULT" | grep -i "error"; then
handle_error "Failed to create S3 bucket: $CREATE_BUCKET_RESULT"
fi
echo "$CREATE_BUCKET_RESULT"
# Step 1: Create a database
echo "Step 1: Creating Athena database: $DATABASE_NAME"
CREATE_DB_RESULT=$(aws athena start-query-execution \
--query-string "CREATE DATABASE IF NOT EXISTS $DATABASE_NAME" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$CREATE_DB_RESULT" | grep -i "error"; then
handle_error "Failed to create database: $CREATE_DB_RESULT"
fi
QUERY_ID=$(echo "$CREATE_DB_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Database creation query ID: $QUERY_ID"
# Wait for database creation to complete
echo "Waiting for database creation to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Database creation completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Database creation failed with status: $QUERY_STATUS"
fi
echo "Database creation in progress, status: $QUERY_STATUS"
sleep 2
done
# Verify the database was created
echo "Verifying database creation..."
LIST_DB_RESULT=$(aws athena list-databases --catalog-name AwsDataCatalog 2>&1)
if echo "$LIST_DB_RESULT" | grep -i "error"; then
handle_error "Failed to list databases: $LIST_DB_RESULT"
fi
echo "$LIST_DB_RESULT"
# Step 2: Create a table
echo "Step 2: Creating Athena table: $TABLE_NAME"
# Replace the region placeholder in the S3 location
CREATE_TABLE_QUERY="CREATE EXTERNAL TABLE IF NOT EXISTS $DATABASE_NAME.$TABLE_NAME (
\`Date\` DATE,
Time STRING,
Location STRING,
Bytes INT,
RequestIP STRING,
Method STRING,
Host STRING,
Uri STRING,
Status INT,
Referrer STRING,
os STRING,
Browser STRING,
BrowserVersion STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
\"input.regex\" = \"^(?!#)([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+[^\\\\(]+[\\\\(]([^\\\\;]+).*\\\\%20([^\\\\/]+)[\\\\/](.*)$\"
) LOCATION 's3://athena-examples-us-east-1/cloudfront/plaintext/';"
CREATE_TABLE_RESULT=$(aws athena start-query-execution \
--query-string "$CREATE_TABLE_QUERY" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$CREATE_TABLE_RESULT" | grep -i "error"; then
handle_error "Failed to create table: $CREATE_TABLE_RESULT"
fi
QUERY_ID=$(echo "$CREATE_TABLE_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Table creation query ID: $QUERY_ID"
# Wait for table creation to complete
echo "Waiting for table creation to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Table creation completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Table creation failed with status: $QUERY_STATUS"
fi
echo "Table creation in progress, status: $QUERY_STATUS"
sleep 2
done
# Verify the table was created
echo "Verifying table creation..."
LIST_TABLE_RESULT=$(aws athena list-table-metadata \
--catalog-name AwsDataCatalog \
--database-name "$DATABASE_NAME" 2>&1)
if echo "$LIST_TABLE_RESULT" | grep -i "error"; then
handle_error "Failed to list tables: $LIST_TABLE_RESULT"
fi
echo "$LIST_TABLE_RESULT"
# Step 3: Query data
echo "Step 3: Running a query on the table..."
QUERY="SELECT os, COUNT(*) count
FROM $DATABASE_NAME.$TABLE_NAME
WHERE date BETWEEN date '2014-07-05' AND date '2014-08-05'
GROUP BY os"
QUERY_RESULT=$(aws athena start-query-execution \
--query-string "$QUERY" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$QUERY_RESULT" | grep -i "error"; then
handle_error "Failed to run query: $QUERY_RESULT"
fi
QUERY_ID=$(echo "$QUERY_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Query execution ID: $QUERY_ID"
# Wait for query to complete
echo "Waiting for query to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Query completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Query failed with status: $QUERY_STATUS"
fi
echo "Query in progress, status: $QUERY_STATUS"
sleep 2
done
# Get query results
echo "Getting query results..."
RESULTS=$(aws athena get-query-results --query-execution-id "$QUERY_ID" 2>&1)
if echo "$RESULTS" | grep -i "error"; then
handle_error "Failed to get query results: $RESULTS"
fi
echo "$RESULTS"
# Download results from S3
echo "Downloading query results from S3..."
S3_PATH=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.ResultConfiguration.OutputLocation" --output text 2>&1)
if echo "$S3_PATH" | grep -i "error"; then
handle_error "Failed to get S3 path for results: $S3_PATH"
fi
DOWNLOAD_RESULT=$(aws s3 cp "$S3_PATH" "./query-results.csv" 2>&1)
if echo "$DOWNLOAD_RESULT" | grep -i "error"; then
handle_error "Failed to download query results: $DOWNLOAD_RESULT"
fi
echo "Query results downloaded to query-results.csv"
# Step 4: Create a named query
echo "Step 4: Creating a named query..."
NAMED_QUERY_RESULT=$(aws athena create-named-query \
--name "OS Count Query" \
--description "Count of operating systems in CloudFront logs" \
--database "$DATABASE_NAME" \
--query-string "$QUERY" 2>&1)
if echo "$NAMED_QUERY_RESULT" | grep -i "error"; then
handle_error "Failed to create named query: $NAMED_QUERY_RESULT"
fi
NAMED_QUERY_ID=$(echo "$NAMED_QUERY_RESULT" | grep -o '"NamedQueryId": "[^"]*' | cut -d'"' -f4)
echo "Named query created with ID: $NAMED_QUERY_ID"
# List named queries
echo "Listing named queries..."
LIST_QUERIES_RESULT=$(aws athena list-named-queries 2>&1)
if echo "$LIST_QUERIES_RESULT" | grep -i "error"; then
handle_error "Failed to list named queries: $LIST_QUERIES_RESULT"
fi
echo "$LIST_QUERIES_RESULT"
# Get the named query details
echo "Getting named query details..."
GET_QUERY_RESULT=$(aws athena get-named-query --named-query-id "$NAMED_QUERY_ID" 2>&1)
if echo "$GET_QUERY_RESULT" | grep -i "error"; then
handle_error "Failed to get named query: $GET_QUERY_RESULT"
fi
echo "$GET_QUERY_RESULT"
# Execute the named query
echo "Executing the named query..."
QUERY_STRING=$(aws athena get-named-query --named-query-id "$NAMED_QUERY_ID" --query "NamedQuery.QueryString" --output text 2>&1)
if echo "$QUERY_STRING" | grep -i "error"; then
handle_error "Failed to get query string: $QUERY_STRING"
fi
EXEC_RESULT=$(aws athena start-query-execution \
--query-string "$QUERY_STRING" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$EXEC_RESULT" | grep -i "error"; then
handle_error "Failed to execute named query: $EXEC_RESULT"
fi
QUERY_ID=$(echo "$EXEC_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Named query execution ID: $QUERY_ID"
# Wait for named query to complete
echo "Waiting for named query execution to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Named query execution completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Named query execution failed with status: $QUERY_STATUS"
fi
echo "Named query execution in progress, status: $QUERY_STATUS"
sleep 2
done
# Summary of resources created
echo ""
echo "==========================================="
echo "RESOURCES CREATED"
echo "==========================================="
echo "- S3 Bucket: $S3_BUCKET"
echo "- Database: $DATABASE_NAME"
echo "- Table: $TABLE_NAME"
echo "- Named Query: $NAMED_QUERY_ID"
echo "- Query results saved to: query-results.csv"
echo "==========================================="
# Prompt for cleanup
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
echo "Starting cleanup..."
# Delete named query
echo "Deleting named query: $NAMED_QUERY_ID"
DELETE_QUERY_RESULT=$(aws athena delete-named-query --named-query-id "$NAMED_QUERY_ID" 2>&1)
if echo "$DELETE_QUERY_RESULT" | grep -i "error"; then
echo "Warning: Failed to delete named query: $DELETE_QUERY_RESULT"
else
echo "Named query deleted successfully."
fi
# Drop table
echo "Dropping table: $TABLE_NAME"
DROP_TABLE_RESULT=$(aws athena start-query-execution \
--query-string "DROP TABLE IF EXISTS $DATABASE_NAME.$TABLE_NAME" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$DROP_TABLE_RESULT" | grep -i "error"; then
echo "Warning: Failed to drop table: $DROP_TABLE_RESULT"
else
QUERY_ID=$(echo "$DROP_TABLE_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Waiting for table deletion to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Table dropped successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
echo "Warning: Table deletion failed with status: $QUERY_STATUS"
break
fi
echo "Table deletion in progress, status: $QUERY_STATUS"
sleep 2
done
fi
# Drop database
echo "Dropping database: $DATABASE_NAME"
DROP_DB_RESULT=$(aws athena start-query-execution \
--query-string "DROP DATABASE IF EXISTS $DATABASE_NAME" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$DROP_DB_RESULT" | grep -i "error"; then
echo "Warning: Failed to drop database: $DROP_DB_RESULT"
else
QUERY_ID=$(echo "$DROP_DB_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Waiting for database deletion to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Database dropped successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
echo "Warning: Database deletion failed with status: $QUERY_STATUS"
break
fi
echo "Database deletion in progress, status: $QUERY_STATUS"
sleep 2
done
fi
# Empty and delete S3 bucket
echo "Emptying S3 bucket: $S3_BUCKET"
EMPTY_BUCKET_RESULT=$(aws s3 rm "s3://$S3_BUCKET" --recursive 2>&1)
if echo "$EMPTY_BUCKET_RESULT" | grep -i "error"; then
echo "Warning: Failed to empty S3 bucket: $EMPTY_BUCKET_RESULT"
else
echo "S3 bucket emptied successfully."
fi
echo "Deleting S3 bucket: $S3_BUCKET"
DELETE_BUCKET_RESULT=$(aws s3 rb "s3://$S3_BUCKET" 2>&1)
if echo "$DELETE_BUCKET_RESULT" | grep -i "error"; then
echo "Warning: Failed to delete S3 bucket: $DELETE_BUCKET_RESULT"
else
echo "S3 bucket deleted successfully."
fi
echo "Cleanup completed."
else
echo "Cleanup skipped. Resources will remain in your AWS account."
fi
echo "Tutorial completed successfully!"
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [CreateNamedQuery](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/CreateNamedQuery)
+ [DeleteNamedQuery](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/DeleteNamedQuery)
+ [GetNamedQuery](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/GetNamedQuery)
+ [GetQueryExecution](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/GetQueryExecution)
+ [GetQueryResults](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/GetQueryResults)
+ [ListDatabases](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/ListDatabases)
+ [ListNamedQueries](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/ListNamedQueries)
+ [ListTableMetadata](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/ListTableMetadata)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [StartQueryExecution](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/StartQueryExecution)
### Getting started with Amazon EMR
The following code example shows how to:
+ Create an EC2 key pair
+ Set up storage and prepare your application
+ Clean up resources
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/037-emr-gs) repository.
```
#!/bin/bash
# EMR Getting Started Tutorial Script
# This script automates the steps in the Amazon EMR Getting Started tutorial
# Set up logging
LOG_FILE="emr-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting Amazon EMR Getting Started Tutorial Script"
echo "Logging to $LOG_FILE"
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Resources created so far:"
if [ -n "$BUCKET_NAME" ]; then echo "- S3 Bucket: $BUCKET_NAME"; fi
if [ -n "$CLUSTER_ID" ]; then echo "- EMR Cluster: $CLUSTER_ID"; fi
echo "Attempting to clean up resources..."
cleanup
exit 1
}
# Function to clean up resources
cleanup() {
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "${CLEANUP_CHOICE,,}" == "y" ]]; then
echo "Starting cleanup process..."
# Terminate EMR cluster if it exists
if [ -n "$CLUSTER_ID" ]; then
echo "Terminating EMR cluster: $CLUSTER_ID"
aws emr terminate-clusters --cluster-ids "$CLUSTER_ID"
echo "Waiting for cluster to terminate..."
aws emr wait cluster-terminated --cluster-id "$CLUSTER_ID"
echo "Cluster terminated successfully."
fi
# Delete S3 bucket and contents if it exists
if [ -n "$BUCKET_NAME" ]; then
echo "Deleting S3 bucket contents: $BUCKET_NAME"
aws s3 rm "s3://$BUCKET_NAME" --recursive
echo "Deleting S3 bucket: $BUCKET_NAME"
aws s3 rb "s3://$BUCKET_NAME"
fi
echo "Cleanup completed."
else
echo "Cleanup skipped. Resources will remain in your AWS account."
echo "To avoid ongoing charges, remember to manually delete these resources."
fi
}
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
BUCKET_NAME="emr${RANDOM_ID}"
echo "Using bucket name: $BUCKET_NAME"
# Create S3 bucket
echo "Creating S3 bucket: $BUCKET_NAME"
aws s3 mb "s3://$BUCKET_NAME" || handle_error "Failed to create S3 bucket"
echo "S3 bucket created successfully."
# Create PySpark script
echo "Creating PySpark script: health_violations.py"
cat > health_violations.py << 'EOL'
import argparse
from pyspark.sql import SparkSession
def calculate_red_violations(data_source, output_uri):
"""
Processes sample food establishment inspection data and queries the data to find the top 10 establishments
with the most Red violations from 2006 to 2020.
:param data_source: The URI of your food establishment data CSV, such as 's3://emr-tutorial-bucket/food-establishment-data.csv'.
:param output_uri: The URI where output is written, such as 's3://emr-tutorial-bucket/restaurant_violation_results'.
"""
with SparkSession.builder.appName("Calculate Red Health Violations").getOrCreate() as spark:
# Load the restaurant violation CSV data
if data_source is not None:
restaurants_df = spark.read.option("header", "true").csv(data_source)
# Create an in-memory DataFrame to query
restaurants_df.createOrReplaceTempView("restaurant_violations")
# Create a DataFrame of the top 10 restaurants with the most Red violations
top_red_violation_restaurants = spark.sql("""SELECT name, count(*) AS total_red_violations
FROM restaurant_violations
WHERE violation_type = 'RED'
GROUP BY name
ORDER BY total_red_violations DESC LIMIT 10""")
# Write the results to the specified output URI
top_red_violation_restaurants.write.option("header", "true").mode("overwrite").csv(output_uri)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
'--data_source', help="The URI for you CSV restaurant data, like an S3 bucket location.")
parser.add_argument(
'--output_uri', help="The URI where output is saved, like an S3 bucket location.")
args = parser.parse_args()
calculate_red_violations(args.data_source, args.output_uri)
EOL
# Upload PySpark script to S3
echo "Uploading PySpark script to S3"
aws s3 cp health_violations.py "s3://$BUCKET_NAME/" || handle_error "Failed to upload PySpark script"
echo "PySpark script uploaded successfully."
# Download and prepare sample data
echo "Downloading sample data"
curl -o food_establishment_data.zip https://docs.aws.amazon.com/emr/latest/ManagementGuide/samples/food_establishment_data.zip || handle_error "Failed to download sample data"
unzip -o food_establishment_data.zip || handle_error "Failed to unzip sample data"
echo "Sample data downloaded and extracted successfully."
# Upload sample data to S3
echo "Uploading sample data to S3"
aws s3 cp food_establishment_data.csv "s3://$BUCKET_NAME/" || handle_error "Failed to upload sample data"
echo "Sample data uploaded successfully."
# Create IAM default roles for EMR
echo "Creating IAM default roles for EMR"
aws emr create-default-roles || handle_error "Failed to create default roles"
echo "IAM default roles created successfully."
# Check if EC2 key pair exists
echo "Checking for EC2 key pair"
KEY_PAIRS=$(aws ec2 describe-key-pairs --query "KeyPairs[*].KeyName" --output text)
if [ -z "$KEY_PAIRS" ]; then
echo "No EC2 key pairs found. Creating a new key pair..."
KEY_NAME="emr-tutorial-key-$RANDOM_ID"
aws ec2 create-key-pair --key-name "$KEY_NAME" --query "KeyMaterial" --output text > "${KEY_NAME}.pem"
chmod 400 "${KEY_NAME}.pem"
echo "Created new key pair: $KEY_NAME"
else
# Use the first available key pair
KEY_NAME=$(echo "$KEY_PAIRS" | awk '{print $1}')
echo "Using existing key pair: $KEY_NAME"
fi
# Launch EMR cluster
echo "Launching EMR cluster with Spark"
CLUSTER_RESPONSE=$(aws emr create-cluster \
--name "EMR Tutorial Cluster" \
--release-label emr-6.10.0 \
--applications Name=Spark \
--ec2-attributes KeyName="$KEY_NAME" \
--instance-type m5.xlarge \
--instance-count 3 \
--use-default-roles \
--log-uri "s3://$BUCKET_NAME/logs/")
# Check for errors in the response
if echo "$CLUSTER_RESPONSE" | grep -i "error" > /dev/null; then
handle_error "Failed to create EMR cluster: $CLUSTER_RESPONSE"
fi
# Extract cluster ID
CLUSTER_ID=$(echo "$CLUSTER_RESPONSE" | grep -o '"ClusterId": "[^"]*' | cut -d'"' -f4)
if [ -z "$CLUSTER_ID" ]; then
handle_error "Failed to extract cluster ID from response"
fi
echo "EMR cluster created with ID: $CLUSTER_ID"
# Wait for cluster to be ready
echo "Waiting for cluster to be ready (this may take several minutes)..."
aws emr wait cluster-running --cluster-id "$CLUSTER_ID" || handle_error "Cluster failed to reach running state"
# Check if cluster is in WAITING state
CLUSTER_STATE=$(aws emr describe-cluster --cluster-id "$CLUSTER_ID" --query "Cluster.Status.State" --output text)
if [ "$CLUSTER_STATE" != "WAITING" ]; then
echo "Waiting for cluster to reach WAITING state..."
while [ "$CLUSTER_STATE" != "WAITING" ]; do
sleep 30
CLUSTER_STATE=$(aws emr describe-cluster --cluster-id "$CLUSTER_ID" --query "Cluster.Status.State" --output text)
echo "Current cluster state: $CLUSTER_STATE"
# Check for error states
if [[ "$CLUSTER_STATE" == "TERMINATED_WITH_ERRORS" || "$CLUSTER_STATE" == "TERMINATED" ]]; then
handle_error "Cluster entered error state: $CLUSTER_STATE"
fi
done
fi
echo "Cluster is now in WAITING state and ready to accept work."
# Submit Spark application as a step
echo "Submitting Spark application as a step"
STEP_RESPONSE=$(aws emr add-steps \
--cluster-id "$CLUSTER_ID" \
--steps Type=Spark,Name="Health Violations Analysis",ActionOnFailure=CONTINUE,Args=["s3://$BUCKET_NAME/health_violations.py","--data_source","s3://$BUCKET_NAME/food_establishment_data.csv","--output_uri","s3://$BUCKET_NAME/results/"])
# Check for errors in the response
if echo "$STEP_RESPONSE" | grep -i "error" > /dev/null; then
handle_error "Failed to submit step: $STEP_RESPONSE"
fi
# FIXED: Check if jq is available before using it
# Extract step ID using the appropriate method based on available tools
if command -v jq &> /dev/null; then
# Use jq if available
echo "Using jq to parse JSON response"
STEP_ID=$(echo "$STEP_RESPONSE" | jq -r '.StepIds[0]')
else
# Fallback to grep/awk if jq is not available
echo "jq not found, using grep for parsing"
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*"' | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
# Another fallback method
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*' | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
# One more attempt with a different pattern
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
echo "Full step response: $STEP_RESPONSE"
handle_error "Failed to extract step ID from response"
fi
fi
fi
fi
if [ -z "$STEP_ID" ] || [ "$STEP_ID" == "null" ]; then
echo "Full step response: $STEP_RESPONSE"
handle_error "Failed to extract valid step ID from response"
fi
echo "Step submitted with ID: $STEP_ID"
# Wait for step to complete
echo "Waiting for step to complete (this may take several minutes)..."
aws emr wait step-complete --cluster-id "$CLUSTER_ID" --step-id "$STEP_ID" || handle_error "Step failed to complete"
# Check step status
STEP_STATE=$(aws emr describe-step --cluster-id "$CLUSTER_ID" --step-id "$STEP_ID" --query "Step.Status.State" --output text)
if [ "$STEP_STATE" != "COMPLETED" ]; then
handle_error "Step did not complete successfully. Final state: $STEP_STATE"
fi
echo "Step completed successfully."
# View results
echo "Listing output files in S3"
aws s3 ls "s3://$BUCKET_NAME/results/" || handle_error "Failed to list output files"
# Download results
echo "Downloading results file"
RESULT_FILE=$(aws s3 ls "s3://$BUCKET_NAME/results/" | grep -o "part-[0-9]*.csv" | head -1)
if [ -z "$RESULT_FILE" ]; then
echo "No result file found with pattern 'part-[0-9]*.csv'. Trying to find any CSV file..."
RESULT_FILE=$(aws s3 ls "s3://$BUCKET_NAME/results/" | grep -o "part-.*\.csv" | head -1)
if [ -z "$RESULT_FILE" ]; then
echo "Listing all files in results directory:"
aws s3 ls "s3://$BUCKET_NAME/results/"
handle_error "No result file found in the output directory"
fi
fi
aws s3 cp "s3://$BUCKET_NAME/results/$RESULT_FILE" ./results.csv || handle_error "Failed to download results file"
echo "Results downloaded to results.csv"
echo "Top 10 establishments with the most red violations:"
cat results.csv
# Display SSH connection information
echo ""
echo "To connect to the cluster via SSH, use the following command:"
echo "aws emr ssh --cluster-id $CLUSTER_ID --key-pair-file ${KEY_NAME}.pem"
# Display summary of created resources
echo ""
echo "==========================================="
echo "RESOURCES CREATED"
echo "==========================================="
echo "- S3 Bucket: $BUCKET_NAME"
echo "- EMR Cluster: $CLUSTER_ID"
echo "- Results file: results.csv"
if [ -f "${KEY_NAME}.pem" ]; then
echo "- EC2 Key Pair: $KEY_NAME (saved to ${KEY_NAME}.pem)"
fi
# Offer to clean up resources
cleanup
echo "Script completed successfully."
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AddSteps](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/AddSteps)
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [CreateCluster](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/CreateCluster)
+ [CreateDefaultRoles](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/CreateDefaultRoles)
+ [CreateKeyPair](https://docs.aws.amazon.com/goto/aws-cli/ec2-2016-11-15/CreateKeyPair)
+ [DescribeCluster](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/DescribeCluster)
+ [DescribeKeyPairs](https://docs.aws.amazon.com/goto/aws-cli/ec2-2016-11-15/DescribeKeyPairs)
+ [DescribeStep](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/DescribeStep)
+ [Ls](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Ls)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [Ssh](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/Ssh)
+ [TerminateClusters](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/TerminateClusters)
+ [Wait](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/Wait)
### Getting started with Amazon S3
The following code example shows how to:
+ Create your first S3 bucket
+ Upload an object
+ Enable versioning
+ Configure default encryption
+ Add tags to your bucket
+ List objects and versions
+ Clean up resources
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/003-s3-gettingstarted) repository.
```
#!/bin/bash
# S3 Getting Started - Create a bucket, upload and download objects, copy to a
# folder prefix, enable versioning, configure encryption and public access
# blocking, tag the bucket, list objects and versions, and clean up.
set -eE
# ============================================================================
# Prerequisites check
# ============================================================================
CONFIGURED_REGION=$(aws configure get region 2>/dev/null || true)
if [ -z "$CONFIGURED_REGION" ] && [ -z "$AWS_DEFAULT_REGION" ] && [ -z "$AWS_REGION" ]; then
echo "ERROR: No AWS region configured. Run 'aws configure' or set AWS_DEFAULT_REGION."
exit 1
fi
# ============================================================================
# Setup: logging, temp directory, resource tracking
# ============================================================================
UNIQUE_ID=$(cat /dev/urandom | tr -dc 'a-f0-9' | fold -w 12 | head -n 1)
BUCKET_NAME="s3api-${UNIQUE_ID}"
TEMP_DIR=$(mktemp -d)
LOG_FILE="${TEMP_DIR}/s3-gettingstarted.log"
CREATED_RESOURCES=()
exec > >(tee -a "$LOG_FILE") 2>&1
echo "============================================"
echo "S3 Getting Started"
echo "============================================"
echo "Bucket name: ${BUCKET_NAME}"
echo "Temp directory: ${TEMP_DIR}"
echo "Log file: ${LOG_FILE}"
echo ""
# ============================================================================
# Error handling and cleanup functions
# ============================================================================
cleanup() {
echo ""
echo "============================================"
echo "CLEANUP"
echo "============================================"
# Delete all object versions and delete markers
echo "Listing all object versions in bucket..."
VERSIONS_OUTPUT=$(aws s3api list-object-versions \
--bucket "$BUCKET_NAME" \
--query "Versions[].{Key:Key,VersionId:VersionId}" \
--output text 2>&1) || true
if [ -n "$VERSIONS_OUTPUT" ] && [ "$VERSIONS_OUTPUT" != "None" ]; then
while IFS=$'\t' read -r KEY VERSION_ID; do
if [ -n "$KEY" ] && [ "$KEY" != "None" ]; then
echo "Deleting version: ${KEY} (${VERSION_ID})"
aws s3api delete-object \
--bucket "$BUCKET_NAME" \
--key "$KEY" \
--version-id "$VERSION_ID" 2>&1 || echo "WARNING: Failed to delete version ${KEY} (${VERSION_ID})"
fi
done <<< "$VERSIONS_OUTPUT"
fi
DELETE_MARKERS_OUTPUT=$(aws s3api list-object-versions \
--bucket "$BUCKET_NAME" \
--query "DeleteMarkers[].{Key:Key,VersionId:VersionId}" \
--output text 2>&1) || true
if [ -n "$DELETE_MARKERS_OUTPUT" ] && [ "$DELETE_MARKERS_OUTPUT" != "None" ]; then
while IFS=$'\t' read -r KEY VERSION_ID; do
if [ -n "$KEY" ] && [ "$KEY" != "None" ]; then
echo "Deleting delete marker: ${KEY} (${VERSION_ID})"
aws s3api delete-object \
--bucket "$BUCKET_NAME" \
--key "$KEY" \
--version-id "$VERSION_ID" 2>&1 || echo "WARNING: Failed to delete marker ${KEY} (${VERSION_ID})"
fi
done <<< "$DELETE_MARKERS_OUTPUT"
fi
echo "Deleting bucket: ${BUCKET_NAME}"
aws s3api delete-bucket --bucket "$BUCKET_NAME" 2>&1 || echo "WARNING: Failed to delete bucket ${BUCKET_NAME}"
echo ""
echo "Cleaning up temp directory: ${TEMP_DIR}"
rm -rf "$TEMP_DIR"
echo ""
echo "Cleanup complete."
}
handle_error() {
echo ""
echo "============================================"
echo "ERROR on $1"
echo "============================================"
echo ""
echo "Resources created before error:"
for RESOURCE in "${CREATED_RESOURCES[@]}"; do
echo " - ${RESOURCE}"
done
echo ""
echo "Attempting cleanup..."
cleanup
exit 1
}
trap 'handle_error "line $LINENO"' ERR
# ============================================================================
# Step 1: Create a bucket
# ============================================================================
echo "Step 1: Creating bucket ${BUCKET_NAME}..."
# CreateBucket requires LocationConstraint for all regions except us-east-1
REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-${CONFIGURED_REGION}}}"
if [ "$REGION" = "us-east-1" ]; then
CREATE_OUTPUT=$(aws s3api create-bucket \
--bucket "$BUCKET_NAME" 2>&1)
else
CREATE_OUTPUT=$(aws s3api create-bucket \
--bucket "$BUCKET_NAME" \
--create-bucket-configuration LocationConstraint="$REGION" 2>&1)
fi
echo "$CREATE_OUTPUT"
CREATED_RESOURCES+=("s3:bucket:${BUCKET_NAME}")
echo "Bucket created."
echo ""
# ============================================================================
# Step 2: Upload a sample text file
# ============================================================================
echo "Step 2: Uploading a sample text file..."
SAMPLE_FILE="${TEMP_DIR}/sample.txt"
echo "Hello, Amazon S3! This is a sample file for the getting started tutorial." > "$SAMPLE_FILE"
UPLOAD_OUTPUT=$(aws s3api put-object \
--bucket "$BUCKET_NAME" \
--key "sample.txt" \
--body "$SAMPLE_FILE" 2>&1)
echo "$UPLOAD_OUTPUT"
echo "File uploaded."
echo ""
# ============================================================================
# Step 3: Download the object
# ============================================================================
echo "Step 3: Downloading the object..."
DOWNLOAD_FILE="${TEMP_DIR}/downloaded-sample.txt"
aws s3api get-object \
--bucket "$BUCKET_NAME" \
--key "sample.txt" \
"$DOWNLOAD_FILE" 2>&1
echo "Downloaded to: ${DOWNLOAD_FILE}"
echo "Contents:"
cat "$DOWNLOAD_FILE"
echo ""
# ============================================================================
# Step 4: Copy the object to a folder prefix
# ============================================================================
echo "Step 4: Copying object to a folder prefix..."
COPY_OUTPUT=$(aws s3api copy-object \
--bucket "$BUCKET_NAME" \
--copy-source "${BUCKET_NAME}/sample.txt" \
--key "backup/sample.txt" 2>&1)
echo "$COPY_OUTPUT"
echo "Object copied to backup/sample.txt."
echo ""
# ============================================================================
# Step 5: Enable versioning and upload a second version
# ============================================================================
echo "Step 5: Enabling versioning..."
VERSIONING_OUTPUT=$(aws s3api put-bucket-versioning \
--bucket "$BUCKET_NAME" \
--versioning-configuration Status=Enabled 2>&1)
echo "$VERSIONING_OUTPUT"
echo "Versioning enabled."
echo "Uploading a second version of sample.txt..."
echo "Hello, Amazon S3! This is version 2 of the sample file." > "$SAMPLE_FILE"
UPLOAD_V2_OUTPUT=$(aws s3api put-object \
--bucket "$BUCKET_NAME" \
--key "sample.txt" \
--body "$SAMPLE_FILE" 2>&1)
echo "$UPLOAD_V2_OUTPUT"
echo "Second version uploaded."
echo ""
# ============================================================================
# Step 6: Configure SSE-S3 encryption
# ============================================================================
echo "Step 6: Configuring SSE-S3 default encryption..."
ENCRYPTION_OUTPUT=$(aws s3api put-bucket-encryption \
--bucket "$BUCKET_NAME" \
--server-side-encryption-configuration '{
"Rules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "AES256"
},
"BucketKeyEnabled": true
}
]
}' 2>&1)
echo "$ENCRYPTION_OUTPUT"
echo "SSE-S3 encryption configured."
echo ""
# ============================================================================
# Step 7: Block all public access
# ============================================================================
echo "Step 7: Blocking all public access..."
PUBLIC_ACCESS_OUTPUT=$(aws s3api put-public-access-block \
--bucket "$BUCKET_NAME" \
--public-access-block-configuration '{
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}' 2>&1)
echo "$PUBLIC_ACCESS_OUTPUT"
echo "Public access blocked."
echo ""
# ============================================================================
# Step 8: Tag the bucket
# ============================================================================
echo "Step 8: Tagging the bucket..."
TAG_OUTPUT=$(aws s3api put-bucket-tagging \
--bucket "$BUCKET_NAME" \
--tagging '{
"TagSet": [
{
"Key": "Environment",
"Value": "Tutorial"
},
{
"Key": "Project",
"Value": "S3-GettingStarted"
}
]
}' 2>&1)
echo "$TAG_OUTPUT"
echo "Bucket tagged."
echo "Verifying tags..."
GET_TAGS_OUTPUT=$(aws s3api get-bucket-tagging \
--bucket "$BUCKET_NAME" 2>&1)
echo "$GET_TAGS_OUTPUT"
echo ""
# ============================================================================
# Step 9: List objects and versions
# ============================================================================
echo "Step 9: Listing objects..."
LIST_OUTPUT=$(aws s3api list-objects-v2 \
--bucket "$BUCKET_NAME" 2>&1)
echo "$LIST_OUTPUT"
echo ""
echo "Listing object versions..."
VERSIONS_LIST=$(aws s3api list-object-versions \
--bucket "$BUCKET_NAME" 2>&1)
echo "$VERSIONS_LIST"
echo ""
# ============================================================================
# Step 10: Cleanup
# ============================================================================
echo ""
echo "============================================"
echo "TUTORIAL COMPLETE"
echo "============================================"
echo ""
echo "Resources created:"
for RESOURCE in "${CREATED_RESOURCES[@]}"; do
echo " - ${RESOURCE}"
done
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
cleanup
else
echo ""
echo "Resources were NOT deleted. To clean up manually, run:"
echo ""
echo " # Delete all object versions"
echo " aws s3api list-object-versions --bucket ${BUCKET_NAME} --query 'Versions[].{Key:Key,VersionId:VersionId}' --output text | while IFS=\$'\\t' read -r KEY VID; do aws s3api delete-object --bucket ${BUCKET_NAME} --key \"\$KEY\" --version-id \"\$VID\"; done"
echo ""
echo " # Delete all delete markers"
echo " aws s3api list-object-versions --bucket ${BUCKET_NAME} --query 'DeleteMarkers[].{Key:Key,VersionId:VersionId}' --output text | while IFS=\$'\\t' read -r KEY VID; do aws s3api delete-object --bucket ${BUCKET_NAME} --key \"\$KEY\" --version-id \"\$VID\"; done"
echo ""
echo " # Delete the bucket"
echo " aws s3api delete-bucket --bucket ${BUCKET_NAME}"
echo ""
echo " # Remove temp directory"
echo " rm -rf ${TEMP_DIR}"
fi
echo ""
echo "Done."
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [CopyObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CopyObject)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteObjects](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteObjects)
+ [GetObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/GetObject)
+ [HeadObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/HeadObject)
+ [ListObjectVersions](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/ListObjectVersions)
+ [ListObjectsV2](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/ListObjectsV2)
+ [PutBucketEncryption](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutBucketEncryption)
+ [PutBucketTagging](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutBucketTagging)
+ [PutBucketVersioning](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutBucketVersioning)
+ [PutObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutObject)
+ [PutPublicAccessBlock](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutPublicAccessBlock)
### Getting started with Amazon SageMaker Feature Store
The following code example shows how to:
+ Set up IAM permissions
+ Create a SageMaker execution role
+ Create feature groups
+ Clean up resources
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/028-sagemaker-featurestore) repository.
```
#!/bin/bash
# Amazon SageMaker Feature Store Tutorial Script - Version 3
# This script demonstrates how to use Amazon SageMaker Feature Store with AWS CLI
# Setup logging
LOG_FILE="sagemaker-featurestore-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting SageMaker Feature Store tutorial script at $(date)"
echo "All commands and outputs will be logged to $LOG_FILE"
echo ""
# Track created resources for cleanup
CREATED_RESOURCES=()
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Attempting to clean up resources..."
cleanup_resources
exit 1
}
# Function to check command status
check_status() {
if echo "$1" | grep -i "error" > /dev/null; then
handle_error "$1"
fi
}
# Function to wait for feature group to be created
wait_for_feature_group() {
local feature_group_name=$1
local status="Creating"
echo "Waiting for feature group ${feature_group_name} to be created..."
while [ "$status" = "Creating" ]; do
sleep 5
status=$(aws sagemaker describe-feature-group \
--feature-group-name "${feature_group_name}" \
--query 'FeatureGroupStatus' \
--output text)
echo "Current status: ${status}"
if [ "$status" = "Failed" ]; then
handle_error "Feature group ${feature_group_name} creation failed"
fi
done
echo "Feature group ${feature_group_name} is now ${status}"
}
# Function to clean up resources
cleanup_resources() {
echo "Cleaning up resources..."
# Clean up in reverse order
for ((i=${#CREATED_RESOURCES[@]}-1; i>=0; i--)); do
resource="${CREATED_RESOURCES[$i]}"
resource_type=$(echo "$resource" | cut -d: -f1)
resource_name=$(echo "$resource" | cut -d: -f2)
echo "Deleting $resource_type: $resource_name"
case "$resource_type" in
"FeatureGroup")
aws sagemaker delete-feature-group --feature-group-name "$resource_name"
;;
"S3Bucket")
echo "Emptying S3 bucket: $resource_name"
aws s3 rm "s3://$resource_name" --recursive 2>/dev/null
echo "Deleting S3 bucket: $resource_name"
aws s3api delete-bucket --bucket "$resource_name" 2>/dev/null
;;
"IAMRole")
echo "Detaching policies from role: $resource_name"
aws iam detach-role-policy --role-name "$resource_name" --policy-arn "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess" 2>/dev/null
aws iam detach-role-policy --role-name "$resource_name" --policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess" 2>/dev/null
echo "Deleting IAM role: $resource_name"
aws iam delete-role --role-name "$resource_name" 2>/dev/null
;;
*)
echo "Unknown resource type: $resource_type"
;;
esac
done
}
# Function to create SageMaker execution role
create_sagemaker_role() {
local role_name="SageMakerFeatureStoreRole-$(openssl rand -hex 4)"
echo "Creating SageMaker execution role: $role_name" >&2
# Create trust policy document
local trust_policy='{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
# Create the role
local role_result=$(aws iam create-role \
--role-name "$role_name" \
--assume-role-policy-document "$trust_policy" \
--description "SageMaker execution role for Feature Store tutorial" 2>&1)
if echo "$role_result" | grep -i "error" > /dev/null; then
handle_error "Failed to create IAM role: $role_result"
fi
echo "Role created successfully" >&2
CREATED_RESOURCES+=("IAMRole:$role_name")
# Attach necessary policies
echo "Attaching policies to role..." >&2
# SageMaker execution policy
local policy1_result=$(aws iam attach-role-policy \
--role-name "$role_name" \
--policy-arn "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess" 2>&1)
if echo "$policy1_result" | grep -i "error" > /dev/null; then
handle_error "Failed to attach SageMaker policy: $policy1_result"
fi
# S3 access policy
local policy2_result=$(aws iam attach-role-policy \
--role-name "$role_name" \
--policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess" 2>&1)
if echo "$policy2_result" | grep -i "error" > /dev/null; then
handle_error "Failed to attach S3 policy: $policy2_result"
fi
# Get account ID for role ARN
local account_id=$(aws sts get-caller-identity --query Account --output text)
local role_arn="arn:aws:iam::${account_id}:role/${role_name}"
echo "Role ARN: $role_arn" >&2
echo "Waiting 10 seconds for role to propagate..." >&2
sleep 10
# Return only the role ARN to stdout
echo "$role_arn"
}
# Handle SageMaker execution role
ROLE_ARN=""
if [ -z "$1" ]; then
echo "Creating SageMaker execution role automatically..."
ROLE_ARN=$(create_sagemaker_role)
if [ -z "$ROLE_ARN" ]; then
handle_error "Failed to create SageMaker execution role"
fi
else
ROLE_ARN="$1"
# Validate the role ARN
ROLE_NAME=$(echo "$ROLE_ARN" | sed 's/.*role\///')
ROLE_CHECK=$(aws iam get-role --role-name "$ROLE_NAME" 2>&1)
if echo "$ROLE_CHECK" | grep -i "error" > /dev/null; then
echo "Creating a new role automatically..."
ROLE_ARN=$(create_sagemaker_role)
if [ -z "$ROLE_ARN" ]; then
handle_error "Failed to create SageMaker execution role"
fi
fi
fi
# Handle cleanup option
AUTO_CLEANUP=""
if [ -n "$2" ]; then
AUTO_CLEANUP="$2"
fi
# Generate a random identifier for resource names
RANDOM_ID=$(openssl rand -hex 4)
echo "Using random identifier: $RANDOM_ID"
# Set variables
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
check_status "$ACCOUNT_ID"
echo "Account ID: $ACCOUNT_ID"
# Get current region
REGION=$(aws configure get region)
if [ -z "$REGION" ]; then
REGION="us-east-1"
echo "No default region configured, using: $REGION"
else
echo "Using region: $REGION"
fi
S3_BUCKET_NAME="sagemaker-featurestore-${RANDOM_ID}-${ACCOUNT_ID}"
PREFIX="featurestore-tutorial"
CURRENT_TIME=$(date +%s)
echo "Creating S3 bucket: $S3_BUCKET_NAME"
# Create bucket in current region
if [ "$REGION" = "us-east-1" ]; then
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME" \
--region "$REGION" 2>&1)
else
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME" \
--region "$REGION" \
--create-bucket-configuration LocationConstraint="$REGION" 2>&1)
fi
if echo "$BUCKET_RESULT" | grep -i "error" > /dev/null; then
echo "Failed to create S3 bucket: $BUCKET_RESULT"
exit 1
fi
echo "$BUCKET_RESULT"
CREATED_RESOURCES+=("S3Bucket:$S3_BUCKET_NAME")
# Block public access to the bucket
BLOCK_RESULT=$(aws s3api put-public-access-block \
--bucket "$S3_BUCKET_NAME" \
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" 2>&1)
if echo "$BLOCK_RESULT" | grep -i "error" > /dev/null; then
echo "Failed to block public access to S3 bucket: $BLOCK_RESULT"
cleanup_resources
exit 1
fi
# Create feature groups
echo "Creating feature groups..."
# Create customers feature group
CUSTOMERS_FEATURE_GROUP_NAME="customers-feature-group-${RANDOM_ID}"
echo "Creating customers feature group: $CUSTOMERS_FEATURE_GROUP_NAME"
CUSTOMERS_RESPONSE=$(aws sagemaker create-feature-group \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record-identifier-feature-name "customer_id" \
--event-time-feature-name "EventTime" \
--feature-definitions '[
{"FeatureName": "customer_id", "FeatureType": "Integral"},
{"FeatureName": "name", "FeatureType": "String"},
{"FeatureName": "age", "FeatureType": "Integral"},
{"FeatureName": "address", "FeatureType": "String"},
{"FeatureName": "membership_type", "FeatureType": "String"},
{"FeatureName": "EventTime", "FeatureType": "Fractional"}
]' \
--online-store-config '{"EnableOnlineStore": true}' \
--offline-store-config '{
"S3StorageConfig": {
"S3Uri": "s3://'${S3_BUCKET_NAME}'/'${PREFIX}'"
},
"DisableGlueTableCreation": false
}' \
--role-arn "$ROLE_ARN" 2>&1)
if echo "$CUSTOMERS_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to create customers feature group: $CUSTOMERS_RESPONSE"
cleanup_resources
exit 1
fi
echo "$CUSTOMERS_RESPONSE"
CREATED_RESOURCES+=("FeatureGroup:$CUSTOMERS_FEATURE_GROUP_NAME")
# Create orders feature group
ORDERS_FEATURE_GROUP_NAME="orders-feature-group-${RANDOM_ID}"
echo "Creating orders feature group: $ORDERS_FEATURE_GROUP_NAME"
ORDERS_RESPONSE=$(aws sagemaker create-feature-group \
--feature-group-name "$ORDERS_FEATURE_GROUP_NAME" \
--record-identifier-feature-name "customer_id" \
--event-time-feature-name "EventTime" \
--feature-definitions '[
{"FeatureName": "customer_id", "FeatureType": "Integral"},
{"FeatureName": "order_id", "FeatureType": "String"},
{"FeatureName": "order_date", "FeatureType": "String"},
{"FeatureName": "product", "FeatureType": "String"},
{"FeatureName": "quantity", "FeatureType": "Integral"},
{"FeatureName": "amount", "FeatureType": "Fractional"},
{"FeatureName": "EventTime", "FeatureType": "Fractional"}
]' \
--online-store-config '{"EnableOnlineStore": true}' \
--offline-store-config '{
"S3StorageConfig": {
"S3Uri": "s3://'${S3_BUCKET_NAME}'/'${PREFIX}'"
},
"DisableGlueTableCreation": false
}' \
--role-arn "$ROLE_ARN" 2>&1)
if echo "$ORDERS_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to create orders feature group: $ORDERS_RESPONSE"
cleanup_resources
exit 1
fi
echo "$ORDERS_RESPONSE"
CREATED_RESOURCES+=("FeatureGroup:$ORDERS_FEATURE_GROUP_NAME")
# Wait for feature groups to be created
wait_for_feature_group "$CUSTOMERS_FEATURE_GROUP_NAME"
wait_for_feature_group "$ORDERS_FEATURE_GROUP_NAME"
# Ingest data into feature groups
echo "Ingesting data into feature groups..."
# Ingest customer data
echo "Ingesting customer data..."
CUSTOMER1_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "573291"},
{"FeatureName": "name", "ValueAsString": "John Doe"},
{"FeatureName": "age", "ValueAsString": "35"},
{"FeatureName": "address", "ValueAsString": "123 Main St"},
{"FeatureName": "membership_type", "ValueAsString": "premium"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$CUSTOMER1_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest customer 1 data: $CUSTOMER1_RESPONSE"
cleanup_resources
exit 1
fi
echo "$CUSTOMER1_RESPONSE"
CUSTOMER2_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "109382"},
{"FeatureName": "name", "ValueAsString": "Jane Smith"},
{"FeatureName": "age", "ValueAsString": "28"},
{"FeatureName": "address", "ValueAsString": "456 Oak Ave"},
{"FeatureName": "membership_type", "ValueAsString": "standard"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$CUSTOMER2_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest customer 2 data: $CUSTOMER2_RESPONSE"
cleanup_resources
exit 1
fi
echo "$CUSTOMER2_RESPONSE"
# Ingest order data
echo "Ingesting order data..."
ORDER1_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$ORDERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "573291"},
{"FeatureName": "order_id", "ValueAsString": "ORD-001"},
{"FeatureName": "order_date", "ValueAsString": "2023-01-15"},
{"FeatureName": "product", "ValueAsString": "Laptop"},
{"FeatureName": "quantity", "ValueAsString": "1"},
{"FeatureName": "amount", "ValueAsString": "1299.99"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$ORDER1_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest order 1 data: $ORDER1_RESPONSE"
cleanup_resources
exit 1
fi
echo "$ORDER1_RESPONSE"
ORDER2_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$ORDERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "109382"},
{"FeatureName": "order_id", "ValueAsString": "ORD-002"},
{"FeatureName": "order_date", "ValueAsString": "2023-01-20"},
{"FeatureName": "product", "ValueAsString": "Smartphone"},
{"FeatureName": "quantity", "ValueAsString": "1"},
{"FeatureName": "amount", "ValueAsString": "899.99"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$ORDER2_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest order 2 data: $ORDER2_RESPONSE"
cleanup_resources
exit 1
fi
echo "$ORDER2_RESPONSE"
# Retrieve records from feature groups
echo "Retrieving records from feature groups..."
# Get a single customer record
echo "Getting customer record with ID 573291:"
CUSTOMER_RECORD=$(aws sagemaker-featurestore-runtime get-record \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record-identifier-value-as-string "573291" 2>&1)
if echo "$CUSTOMER_RECORD" | grep -i "error" > /dev/null; then
echo "Failed to get customer record: $CUSTOMER_RECORD"
cleanup_resources
exit 1
fi
echo "$CUSTOMER_RECORD"
# Get multiple records using batch-get-record
echo "Getting multiple records using batch-get-record:"
BATCH_RECORDS=$(aws sagemaker-featurestore-runtime batch-get-record \
--identifiers '[
{
"FeatureGroupName": "'${CUSTOMERS_FEATURE_GROUP_NAME}'",
"RecordIdentifiersValueAsString": ["573291", "109382"]
},
{
"FeatureGroupName": "'${ORDERS_FEATURE_GROUP_NAME}'",
"RecordIdentifiersValueAsString": ["573291", "109382"]
}
]' 2>&1)
if echo "$BATCH_RECORDS" | grep -i "error" > /dev/null && ! echo "$BATCH_RECORDS" | grep -i "Records" > /dev/null; then
echo "Failed to get batch records: $BATCH_RECORDS"
cleanup_resources
exit 1
fi
echo "$BATCH_RECORDS"
# List feature groups
echo "Listing feature groups:"
FEATURE_GROUPS=$(aws sagemaker list-feature-groups 2>&1)
if echo "$FEATURE_GROUPS" | grep -i "error" > /dev/null; then
echo "Failed to list feature groups: $FEATURE_GROUPS"
cleanup_resources
exit 1
fi
echo "$FEATURE_GROUPS"
# Display summary of created resources
echo ""
echo "==========================================="
echo "TUTORIAL COMPLETED SUCCESSFULLY!"
echo "==========================================="
echo "Resources created:"
echo "- S3 Bucket: $S3_BUCKET_NAME"
echo "- Customers Feature Group: $CUSTOMERS_FEATURE_GROUP_NAME"
echo "- Orders Feature Group: $ORDERS_FEATURE_GROUP_NAME"
if [[ " ${CREATED_RESOURCES[@]} " =~ " IAMRole:" ]]; then
echo "- IAM Role: $(echo "${CREATED_RESOURCES[@]}" | grep -o 'IAMRole:[^[:space:]]*' | cut -d: -f2)"
fi
echo ""
echo "You can now:"
echo "1. View your feature groups in the SageMaker console"
echo "2. Query the offline store using Amazon Athena"
echo "3. Use the feature groups in your ML workflows"
echo "==========================================="
echo ""
# Handle cleanup
if [ "$AUTO_CLEANUP" = "y" ]; then
echo "Auto-cleanup enabled. Starting cleanup..."
cleanup_resources
echo "Cleanup completed."
elif [ "$AUTO_CLEANUP" = "n" ]; then
echo "Auto-cleanup disabled. Resources will remain in your account."
echo "To clean up later, run this script again with cleanup option 'y'"
else
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
echo "Starting cleanup..."
cleanup_resources
echo "Cleanup completed."
else
echo "Skipping cleanup. Resources will remain in your account."
echo "To clean up later, delete the following resources:"
echo "- Feature Groups: $CUSTOMERS_FEATURE_GROUP_NAME, $ORDERS_FEATURE_GROUP_NAME"
echo "- S3 Bucket: $S3_BUCKET_NAME"
if [[ " ${CREATED_RESOURCES[@]} " =~ " IAMRole:" ]]; then
echo "- IAM Role: $(echo "${CREATED_RESOURCES[@]}" | grep -o 'IAMRole:[^[:space:]]*' | cut -d: -f2)"
fi
echo ""
echo "Estimated ongoing cost: ~$0.01 per month for online store"
fi
fi
echo "Script completed at $(date)"
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AttachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/AttachRolePolicy)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [CreateFeatureGroup](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/CreateFeatureGroup)
+ [CreateRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/CreateRole)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteFeatureGroup](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/DeleteFeatureGroup)
+ [DeleteRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DeleteRole)
+ [DescribeFeatureGroup](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/DescribeFeatureGroup)
+ [DetachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DetachRolePolicy)
+ [GetCallerIdentity](https://docs.aws.amazon.com/goto/aws-cli/sts-2011-06-15/GetCallerIdentity)
+ [GetRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/GetRole)
+ [ListFeatureGroups](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/ListFeatureGroups)
+ [PutPublicAccessBlock](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutPublicAccessBlock)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
### Getting started with Amazon Textract
The following code example shows how to:
+ Create an S3 bucket
+ Upload a document to S3
+ Clean up resources
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/074-amazon-textract-gs) repository.
```
#!/bin/bash
# Amazon Textract Getting Started Tutorial Script
# This script demonstrates how to use Amazon Textract to analyze document text
# Set up logging
LOG_FILE="textract-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "==================================================="
echo "Amazon Textract Getting Started Tutorial"
echo "==================================================="
echo "This script will guide you through using Amazon Textract to analyze document text."
echo ""
# Function to check for errors in command output and exit code
check_error() {
local exit_code=$1
local output=$2
local cmd=$3
if [ $exit_code -ne 0 ] || echo "$output" | grep -i "error" > /dev/null; then
echo "ERROR: Command failed: $cmd"
echo "$output"
cleanup_on_error
exit 1
fi
}
# Function to clean up resources on error
cleanup_on_error() {
echo "Error encountered. Cleaning up resources..."
# Clean up temporary JSON files
if [ -f "document.json" ]; then
rm -f document.json
fi
if [ -f "features.json" ]; then
rm -f features.json
fi
if [ -n "$DOCUMENT_NAME" ] && [ -n "$BUCKET_NAME" ]; then
echo "Deleting document from S3..."
aws s3 rm "s3://$BUCKET_NAME/$DOCUMENT_NAME" || echo "Failed to delete document"
fi
if [ -n "$BUCKET_NAME" ]; then
echo "Deleting S3 bucket..."
aws s3 rb "s3://$BUCKET_NAME" --force || echo "Failed to delete bucket"
fi
}
# Verify AWS CLI is installed and configured
echo "Verifying AWS CLI configuration..."
AWS_CONFIG_OUTPUT=$(aws configure list 2>&1)
AWS_CONFIG_STATUS=$?
if [ $AWS_CONFIG_STATUS -ne 0 ]; then
echo "ERROR: AWS CLI is not properly configured."
echo "$AWS_CONFIG_OUTPUT"
exit 1
fi
# Verify AWS region is configured and supports Textract
AWS_REGION=$(aws configure get region)
if [ -z "$AWS_REGION" ]; then
echo "ERROR: No AWS region configured. Please run 'aws configure' to set a default region."
exit 1
fi
# Check if Textract is available in the configured region
echo "Checking if Amazon Textract is available in region $AWS_REGION..."
TEXTRACT_CHECK=$(aws textract help 2>&1)
TEXTRACT_CHECK_STATUS=$?
if [ $TEXTRACT_CHECK_STATUS -ne 0 ]; then
echo "ERROR: Amazon Textract may not be available in region $AWS_REGION."
echo "$TEXTRACT_CHECK"
exit 1
fi
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
BUCKET_NAME="textract-${RANDOM_ID}"
DOCUMENT_NAME="document.png"
RESOURCES_CREATED=()
# Step 1: Create S3 bucket
echo "Creating S3 bucket: $BUCKET_NAME"
CREATE_BUCKET_OUTPUT=$(aws s3 mb "s3://$BUCKET_NAME" 2>&1)
CREATE_BUCKET_STATUS=$?
echo "$CREATE_BUCKET_OUTPUT"
check_error $CREATE_BUCKET_STATUS "$CREATE_BUCKET_OUTPUT" "aws s3 mb s3://$BUCKET_NAME"
RESOURCES_CREATED+=("S3 Bucket: $BUCKET_NAME")
# Step 2: Check if sample document exists, if not create a simple one
if [ ! -f "$DOCUMENT_NAME" ]; then
echo "Sample document not found. Please provide a document to analyze."
echo "Enter the path to your document (must be an image file like PNG or JPEG):"
read -r DOCUMENT_PATH
if [ ! -f "$DOCUMENT_PATH" ]; then
echo "File not found: $DOCUMENT_PATH"
cleanup_on_error
exit 1
fi
DOCUMENT_NAME=$(basename "$DOCUMENT_PATH")
echo "Using document: $DOCUMENT_PATH as $DOCUMENT_NAME"
# Copy the document to the current directory
cp "$DOCUMENT_PATH" "./$DOCUMENT_NAME"
fi
# Step 3: Upload document to S3
echo "Uploading document to S3..."
UPLOAD_OUTPUT=$(aws s3 cp "./$DOCUMENT_NAME" "s3://$BUCKET_NAME/" 2>&1)
UPLOAD_STATUS=$?
echo "$UPLOAD_OUTPUT"
check_error $UPLOAD_STATUS "$UPLOAD_OUTPUT" "aws s3 cp ./$DOCUMENT_NAME s3://$BUCKET_NAME/"
RESOURCES_CREATED+=("S3 Object: s3://$BUCKET_NAME/$DOCUMENT_NAME")
# Step 4: Analyze document with Amazon Textract
echo "Analyzing document with Amazon Textract..."
echo "This may take a few seconds..."
# Create a JSON file for the document parameter to avoid shell escaping issues
cat > document.json << EOF
{
"S3Object": {
"Bucket": "$BUCKET_NAME",
"Name": "$DOCUMENT_NAME"
}
}
EOF
# Create a JSON file for the feature types parameter
cat > features.json << EOF
["TABLES","FORMS","SIGNATURES"]
EOF
ANALYZE_OUTPUT=$(aws textract analyze-document --document file://document.json --feature-types file://features.json 2>&1)
ANALYZE_STATUS=$?
echo "Analysis complete."
if [ $ANALYZE_STATUS -ne 0 ]; then
echo "ERROR: Document analysis failed"
echo "$ANALYZE_OUTPUT"
cleanup_on_error
exit 1
fi
# Save the analysis results to a file
echo "$ANALYZE_OUTPUT" > textract-analysis-results.json
echo "Analysis results saved to textract-analysis-results.json"
RESOURCES_CREATED+=("Local file: textract-analysis-results.json")
# Display a summary of the analysis
echo ""
echo "==================================================="
echo "Analysis Summary"
echo "==================================================="
PAGES=$(echo "$ANALYZE_OUTPUT" | grep -o '"Pages": [0-9]*' | awk '{print $2}')
echo "Document pages: $PAGES"
BLOCKS_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType":' | wc -l)
echo "Total blocks detected: $BLOCKS_COUNT"
# Count different block types
PAGE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "PAGE"' | wc -l)
LINE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "LINE"' | wc -l)
WORD_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "WORD"' | wc -l)
TABLE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "TABLE"' | wc -l)
CELL_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "CELL"' | wc -l)
KEY_VALUE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "KEY_VALUE_SET"' | wc -l)
SIGNATURE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "SIGNATURE"' | wc -l)
echo "Pages: $PAGE_COUNT"
echo "Lines of text: $LINE_COUNT"
echo "Words: $WORD_COUNT"
echo "Tables: $TABLE_COUNT"
echo "Table cells: $CELL_COUNT"
echo "Key-value pairs: $KEY_VALUE_COUNT"
echo "Signatures: $SIGNATURE_COUNT"
echo ""
# Cleanup confirmation
echo ""
echo "==================================================="
echo "RESOURCES CREATED"
echo "==================================================="
for resource in "${RESOURCES_CREATED[@]}"; do
echo "- $resource"
done
echo ""
echo "==================================================="
echo "CLEANUP CONFIRMATION"
echo "==================================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy] ]]; then
echo "Cleaning up resources..."
# Delete document from S3
echo "Deleting document from S3..."
DELETE_DOC_OUTPUT=$(aws s3 rm "s3://$BUCKET_NAME/$DOCUMENT_NAME" 2>&1)
DELETE_DOC_STATUS=$?
echo "$DELETE_DOC_OUTPUT"
check_error $DELETE_DOC_STATUS "$DELETE_DOC_OUTPUT" "aws s3 rm s3://$BUCKET_NAME/$DOCUMENT_NAME"
# Delete S3 bucket
echo "Deleting S3 bucket..."
DELETE_BUCKET_OUTPUT=$(aws s3 rb "s3://$BUCKET_NAME" --force 2>&1)
DELETE_BUCKET_STATUS=$?
echo "$DELETE_BUCKET_OUTPUT"
check_error $DELETE_BUCKET_STATUS "$DELETE_BUCKET_OUTPUT" "aws s3 rb s3://$BUCKET_NAME --force"
# Delete local JSON files
rm -f document.json features.json
echo "Cleanup complete. The analysis results file (textract-analysis-results.json) has been kept."
else
echo "Resources have been preserved."
fi
echo ""
echo "==================================================="
echo "Tutorial complete!"
echo "==================================================="
echo "You have successfully analyzed a document using Amazon Textract."
echo "The analysis results are available in textract-analysis-results.json"
echo ""
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AnalyzeDocument](https://docs.aws.amazon.com/goto/aws-cli/textract-2018-06-27/AnalyzeDocument)
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [Help](https://docs.aws.amazon.com/goto/aws-cli/textract-2018-06-27/Help)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
### Getting started with Config
The following code example shows how to:
+ Create an Amazon S3 bucket
+ Create an Amazon SNS topic
+ Create an IAM role for Config
+ Set up the Config configuration recorder
+ Set up the Config delivery channel
+ Start the configuration recorder
+ Verify the Config setup
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/053-aws-config-gs) repository.
```
#!/bin/bash
# AWS Config Setup Script (v2)
# This script sets up AWS Config with the AWS CLI
# Error handling
set -e
LOGFILE="aws-config-setup-v2.log"
touch $LOGFILE
exec > >(tee -a $LOGFILE)
exec 2>&1
# Function to handle errors
handle_error() {
echo "ERROR: An error occurred at line $1"
echo "Attempting to clean up resources..."
cleanup_resources
exit 1
}
# Set trap for error handling
trap 'handle_error $LINENO' ERR
# Function to generate random identifier
generate_random_id() {
echo $(openssl rand -hex 6)
}
# Function to check if command was successful
check_command() {
if echo "$1" | grep -i "error" > /dev/null; then
echo "ERROR: $1"
return 1
fi
return 0
}
# Function to clean up resources
cleanup_resources() {
if [ -n "$CONFIG_RECORDER_NAME" ]; then
echo "Stopping configuration recorder..."
aws configservice stop-configuration-recorder --configuration-recorder-name "$CONFIG_RECORDER_NAME" 2>/dev/null || true
fi
# Check if we created a new delivery channel before trying to delete it
if [ -n "$DELIVERY_CHANNEL_NAME" ] && [ "$CREATED_NEW_DELIVERY_CHANNEL" = "true" ]; then
echo "Deleting delivery channel..."
aws configservice delete-delivery-channel --delivery-channel-name "$DELIVERY_CHANNEL_NAME" 2>/dev/null || true
fi
if [ -n "$CONFIG_RECORDER_NAME" ] && [ "$CREATED_NEW_CONFIG_RECORDER" = "true" ]; then
echo "Deleting configuration recorder..."
aws configservice delete-configuration-recorder --configuration-recorder-name "$CONFIG_RECORDER_NAME" 2>/dev/null || true
fi
if [ -n "$ROLE_NAME" ]; then
if [ -n "$POLICY_NAME" ]; then
echo "Detaching custom policy from role..."
aws iam delete-role-policy --role-name "$ROLE_NAME" --policy-name "$POLICY_NAME" 2>/dev/null || true
fi
if [ -n "$MANAGED_POLICY_ARN" ]; then
echo "Detaching managed policy from role..."
aws iam detach-role-policy --role-name "$ROLE_NAME" --policy-arn "$MANAGED_POLICY_ARN" 2>/dev/null || true
fi
echo "Deleting IAM role..."
aws iam delete-role --role-name "$ROLE_NAME" 2>/dev/null || true
fi
if [ -n "$SNS_TOPIC_ARN" ]; then
echo "Deleting SNS topic..."
aws sns delete-topic --topic-arn "$SNS_TOPIC_ARN" 2>/dev/null || true
fi
if [ -n "$S3_BUCKET_NAME" ]; then
echo "Emptying S3 bucket..."
aws s3 rm "s3://$S3_BUCKET_NAME" --recursive 2>/dev/null || true
echo "Deleting S3 bucket..."
aws s3api delete-bucket --bucket "$S3_BUCKET_NAME" 2>/dev/null || true
fi
}
# Function to display created resources
display_resources() {
echo ""
echo "==========================================="
echo "CREATED RESOURCES"
echo "==========================================="
echo "S3 Bucket: $S3_BUCKET_NAME"
echo "SNS Topic ARN: $SNS_TOPIC_ARN"
echo "IAM Role: $ROLE_NAME"
if [ "$CREATED_NEW_CONFIG_RECORDER" = "true" ]; then
echo "Configuration Recorder: $CONFIG_RECORDER_NAME (newly created)"
else
echo "Configuration Recorder: $CONFIG_RECORDER_NAME (existing)"
fi
if [ "$CREATED_NEW_DELIVERY_CHANNEL" = "true" ]; then
echo "Delivery Channel: $DELIVERY_CHANNEL_NAME (newly created)"
else
echo "Delivery Channel: $DELIVERY_CHANNEL_NAME (existing)"
fi
echo "==========================================="
}
# Get AWS account ID
echo "Getting AWS account ID..."
ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
if [ -z "$ACCOUNT_ID" ]; then
echo "ERROR: Failed to get AWS account ID"
exit 1
fi
echo "AWS Account ID: $ACCOUNT_ID"
# Generate random identifier for resources
RANDOM_ID=$(generate_random_id)
echo "Generated random identifier: $RANDOM_ID"
# Step 1: Create an S3 bucket
S3_BUCKET_NAME="configservice-${RANDOM_ID}"
echo "Creating S3 bucket: $S3_BUCKET_NAME"
# Get the current region
AWS_REGION=$(aws configure get region)
if [ -z "$AWS_REGION" ]; then
AWS_REGION="us-east-1" # Default to us-east-1 if no region is configured
fi
echo "Using AWS Region: $AWS_REGION"
# Create bucket with appropriate command based on region
if [ "$AWS_REGION" = "us-east-1" ]; then
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME")
else
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME" --create-bucket-configuration LocationConstraint="$AWS_REGION")
fi
check_command "$BUCKET_RESULT"
echo "S3 bucket created: $S3_BUCKET_NAME"
# Block public access for the bucket
aws s3api put-public-access-block \
--bucket "$S3_BUCKET_NAME" \
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
echo "Public access blocked for bucket"
# Step 2: Create an SNS topic
TOPIC_NAME="config-topic-${RANDOM_ID}"
echo "Creating SNS topic: $TOPIC_NAME"
SNS_RESULT=$(aws sns create-topic --name "$TOPIC_NAME")
check_command "$SNS_RESULT"
SNS_TOPIC_ARN=$(echo "$SNS_RESULT" | grep -o 'arn:aws:sns:[^"]*')
echo "SNS topic created: $SNS_TOPIC_ARN"
# Step 3: Create an IAM role for AWS Config
ROLE_NAME="config-role-${RANDOM_ID}"
POLICY_NAME="config-delivery-permissions"
MANAGED_POLICY_ARN="arn:aws:iam::aws:policy/service-role/AWS_ConfigRole"
echo "Creating trust policy document..."
cat > config-trust-policy.json << EOF
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "config.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
echo "Creating IAM role: $ROLE_NAME"
ROLE_RESULT=$(aws iam create-role --role-name "$ROLE_NAME" --assume-role-policy-document file://config-trust-policy.json)
check_command "$ROLE_RESULT"
ROLE_ARN=$(echo "$ROLE_RESULT" | grep -o 'arn:aws:iam::[^"]*' | head -1)
echo "IAM role created: $ROLE_ARN"
echo "Attaching AWS managed policy to role..."
ATTACH_RESULT=$(aws iam attach-role-policy --role-name "$ROLE_NAME" --policy-arn "$MANAGED_POLICY_ARN")
check_command "$ATTACH_RESULT"
echo "AWS managed policy attached"
echo "Creating custom policy document for S3 and SNS access..."
cat > config-delivery-permissions.json << EOF
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::${S3_BUCKET_NAME}/AWSLogs/${ACCOUNT_ID}/*",
"Condition": {
"StringLike": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetBucketAcl"
],
"Resource": "arn:aws:s3:::${S3_BUCKET_NAME}"
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "${SNS_TOPIC_ARN}"
}
]
}
EOF
echo "Attaching custom policy to role..."
POLICY_RESULT=$(aws iam put-role-policy --role-name "$ROLE_NAME" --policy-name "$POLICY_NAME" --policy-document file://config-delivery-permissions.json)
check_command "$POLICY_RESULT"
echo "Custom policy attached"
# Wait for IAM role to propagate
echo "Waiting for IAM role to propagate (15 seconds)..."
sleep 15
# Step 4: Check if configuration recorder already exists
CONFIG_RECORDER_NAME="default"
CREATED_NEW_CONFIG_RECORDER="false"
echo "Checking for existing configuration recorder..."
EXISTING_RECORDERS=$(aws configservice describe-configuration-recorders 2>/dev/null || echo "")
if echo "$EXISTING_RECORDERS" | grep -q "name"; then
echo "Configuration recorder already exists. Will update it."
# Get the name of the existing recorder
CONFIG_RECORDER_NAME=$(echo "$EXISTING_RECORDERS" | grep -o '"name": "[^"]*"' | head -1 | cut -d'"' -f4)
echo "Using existing configuration recorder: $CONFIG_RECORDER_NAME"
else
echo "No existing configuration recorder found. Will create a new one."
CREATED_NEW_CONFIG_RECORDER="true"
fi
echo "Creating configuration recorder configuration..."
cat > configurationRecorder.json << EOF
{
"name": "${CONFIG_RECORDER_NAME}",
"roleARN": "${ROLE_ARN}",
"recordingMode": {
"recordingFrequency": "CONTINUOUS"
}
}
EOF
echo "Creating recording group configuration..."
cat > recordingGroup.json << EOF
{
"allSupported": true,
"includeGlobalResourceTypes": true
}
EOF
echo "Setting up configuration recorder..."
RECORDER_RESULT=$(aws configservice put-configuration-recorder --configuration-recorder file://configurationRecorder.json --recording-group file://recordingGroup.json)
check_command "$RECORDER_RESULT"
echo "Configuration recorder set up"
# Step 5: Check if delivery channel already exists
DELIVERY_CHANNEL_NAME="default"
CREATED_NEW_DELIVERY_CHANNEL="false"
echo "Checking for existing delivery channel..."
EXISTING_CHANNELS=$(aws configservice describe-delivery-channels 2>/dev/null || echo "")
if echo "$EXISTING_CHANNELS" | grep -q "name"; then
echo "Delivery channel already exists."
# Get the name of the existing channel
DELIVERY_CHANNEL_NAME=$(echo "$EXISTING_CHANNELS" | grep -o '"name": "[^"]*"' | head -1 | cut -d'"' -f4)
echo "Using existing delivery channel: $DELIVERY_CHANNEL_NAME"
# Update the existing delivery channel
echo "Creating delivery channel configuration for update..."
cat > deliveryChannel.json << EOF
{
"name": "${DELIVERY_CHANNEL_NAME}",
"s3BucketName": "${S3_BUCKET_NAME}",
"snsTopicARN": "${SNS_TOPIC_ARN}",
"configSnapshotDeliveryProperties": {
"deliveryFrequency": "Six_Hours"
}
}
EOF
echo "Updating delivery channel..."
CHANNEL_RESULT=$(aws configservice put-delivery-channel --delivery-channel file://deliveryChannel.json)
check_command "$CHANNEL_RESULT"
echo "Delivery channel updated"
else
echo "No existing delivery channel found. Will create a new one."
CREATED_NEW_DELIVERY_CHANNEL="true"
echo "Creating delivery channel configuration..."
cat > deliveryChannel.json << EOF
{
"name": "${DELIVERY_CHANNEL_NAME}",
"s3BucketName": "${S3_BUCKET_NAME}",
"snsTopicARN": "${SNS_TOPIC_ARN}",
"configSnapshotDeliveryProperties": {
"deliveryFrequency": "Six_Hours"
}
}
EOF
echo "Creating delivery channel..."
CHANNEL_RESULT=$(aws configservice put-delivery-channel --delivery-channel file://deliveryChannel.json)
check_command "$CHANNEL_RESULT"
echo "Delivery channel created"
fi
# Step 6: Start the configuration recorder
echo "Checking configuration recorder status..."
RECORDER_STATUS=$(aws configservice describe-configuration-recorder-status 2>/dev/null || echo "")
if echo "$RECORDER_STATUS" | grep -q '"recording": true'; then
echo "Configuration recorder is already running."
else
echo "Starting configuration recorder..."
START_RESULT=$(aws configservice start-configuration-recorder --configuration-recorder-name "$CONFIG_RECORDER_NAME")
check_command "$START_RESULT"
echo "Configuration recorder started"
fi
# Step 7: Verify the AWS Config setup
echo "Verifying delivery channel..."
VERIFY_CHANNEL=$(aws configservice describe-delivery-channels)
check_command "$VERIFY_CHANNEL"
echo "$VERIFY_CHANNEL"
echo "Verifying configuration recorder..."
VERIFY_RECORDER=$(aws configservice describe-configuration-recorders)
check_command "$VERIFY_RECORDER"
echo "$VERIFY_RECORDER"
echo "Verifying configuration recorder status..."
VERIFY_STATUS=$(aws configservice describe-configuration-recorder-status)
check_command "$VERIFY_STATUS"
echo "$VERIFY_STATUS"
# Display created resources
display_resources
# Ask if user wants to clean up resources
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
echo "Cleaning up resources..."
cleanup_resources
echo "Cleanup completed."
else
echo "Resources will not be cleaned up. You can manually clean them up later."
fi
echo "Script completed successfully!"
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AttachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/AttachRolePolicy)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [CreateRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/CreateRole)
+ [CreateTopic](https://docs.aws.amazon.com/goto/aws-cli/sns-2010-03-31/CreateTopic)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DeleteConfigurationRecorder)
+ [DeleteDeliveryChannel](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DeleteDeliveryChannel)
+ [DeleteRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DeleteRole)
+ [DeleteRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DeleteRolePolicy)
+ [DeleteTopic](https://docs.aws.amazon.com/goto/aws-cli/sns-2010-03-31/DeleteTopic)
+ [DescribeConfigurationRecorderStatus](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DescribeConfigurationRecorderStatus)
+ [DescribeConfigurationRecorders](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DescribeConfigurationRecorders)
+ [DescribeDeliveryChannels](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DescribeDeliveryChannels)
+ [DetachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DetachRolePolicy)
+ [GetCallerIdentity](https://docs.aws.amazon.com/goto/aws-cli/sts-2011-06-15/GetCallerIdentity)
+ [PutConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/PutConfigurationRecorder)
+ [PutDeliveryChannel](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/PutDeliveryChannel)
+ [PutPublicAccessBlock](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutPublicAccessBlock)
+ [PutRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/PutRolePolicy)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [StartConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/StartConfigurationRecorder)
+ [StopConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/StopConfigurationRecorder)