Configuring transfers with Microsoft Azure Blob Storage

To download and prepare your agent

1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

2. In the left navigation pane, choose Agents, and then choose Create agent.

3. For Hypervisor, choose Microsoft Hyper-V, and then choose Download the image.

   The agent downloads in a .zip file that contains a .vhdx file.

4. Extract the .vhdx file on your local machine.

5. Open PowerShell and do the following:

   1. Copy the following Convert-VHD cmdlet:

      Convert-VHD -Path .\local-path-to-vhdx-file\aws-datasync-2.0.1686143940.1-x86_64.xfs.gpt.vhdx `
      -DestinationPath .\local-path-to-vhdx-file\aws-datasync-2016861439401-x86_64.vhd -VHDType Fixed

   2. Replace each instance of local-path-to-vhdx-file with the location of the .vhdx file on your local machine.

   3. Run the command.

   Your agent is now a fixed-size VHD (with a .vhd file format) and ready to deploy in Azure.

To deploy your agent in Azure

1. In PowerShell, go to the directory that contains your agent's .vhd file.

2. Run the ls command and save the Length value (for example, 85899346432).

   This is the size of your agent image in bytes, which you need when creating a managed disk that can hold the image.

3. Do the following to create a managed disk:

   1. Copy the following Azure CLI command:

      az disk create -n your-managed-disk `
      -g your-resource-group `
      -l your-azure-region `
      --upload-type Upload `
      --upload-size-bytes agent-size-bytes `
      --sku standard_lrs

   2. Replace your-managed-disk with a name for your managed disk.

   3. Replace your-resource-group with the name of the Azure resource group that your storage account belongs to.

   4. Replace your-azure-region with the Azure region where your resource group is located.

   5. Replace agent-size-bytes with the size of your agent image.

   6. Run the command.

   This command creates an empty managed disk with a standard SKU where you can upload your DataSync agent.

4. To generate a shared access signature (SAS) that allows write access to the managed disk, do the following:

   1. Copy the following Azure CLI command:

      az disk grant-access -n your-managed-disk `
      -g your-resource-group `
      --access-level Write `
      --duration-in-seconds 86400

   2. Replace your-managed-disk with the name of the managed disk that you created.

   3. Replace your-resource-group with the name of the Azure resource group that your storage account belongs to.

   4. Run the command.

      In the output, take note of the SAS URI. You need this URI when uploading the agent to Azure.

   The SAS allows you to write to the disk for up to an hour. This means that you have an hour to upload your agent to the managed disk.

5. To upload your agent to your managed disk in Azure, do the following:

   1. Copy the following AzCopy command:

      .\azcopy copy local-path-to-vhd-file sas-uri --blob-type PageBlob

   2. Replace local-path-to-vhd-file with the location of the agent's .vhd file on your local machine.

   3. Replace sas-uri with the SAS URI that you got when you ran the az disk grant-access command.

   4. Run the command.

6. When the agent upload finishes, revoke access to your managed disk. To do this, copy the following Azure CLI command:

   az disk revoke-access -n your-managed-disk -g your-resource-group

   1. Replace your-resource-group with the name of the Azure resource group that your storage account belongs to.

   2. Replace your-managed-disk with the name of the managed disk that you created.

   3. Run the command.

7. Do the following to attach your managed disk to a new Linux VM:

   1. Copy the following Azure CLI command:

      az vm create --resource-group your-resource-group `
      --location eastus `
      --name your-agent-vm `
      --size Standard_E4as_v4 `
      --os-type linux `
      --attach-os-disk your-managed-disk

   2. Replace your-resource-group with the name of the Azure resource group that your storage account belongs to.

   3. Replace your-agent-vm with a name for the VM that you can remember.

   4. Replace your-managed-disk with the name of the managed disk that you're attaching to the VM.

   5. Run the command.

To get your agent's activation key

1. In the Azure portal, enable boot diagnostics for the VM for your agent by choosing the Enable with custom storage account setting and specifying your Azure storage account.

   After you've enabled the boot diagnostics for your agent's VM, you can access your agent’s local console to get the activation key.

2. While still in the Azure portal, go to your VM and choose Serial console.

3. In the agent's local console, log in by using the following default credentials:

   • Username – admin

   • Password – password

   We recommend at some point changing at least the agent's password. In the agent's local console, enter 5 on the main menu, then use the passwd command to change the password.

4. Enter 0 to get the agent's activation key.

5. Enter the AWS Region where you're using DataSync (for example, us-east-1).

6. Choose the service endpoint that the agent will use to connect with AWS.

7. Save the value of the Activation key output.

To activate your agent

1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

2. In the left navigation pane, choose Agents, and then choose Create agent.

3. For Hypervisor, choose Microsoft Hyper-V.

4. For Endpoint type, choose the same type of service endpoint that you specified when you got your agent's activation key (for example, choose Public service endpoints in Region name).

5. Configure your network to work with the service endpoint type that your agent is using. For service endpoint network requirements, see the following topics:

   • VPC endpoints

   • Public endpoints

   • Federal Information Processing Standard (FIPS) endpoints

6. For Activation key, do the following:

   1. Choose Manually enter your agent's activation key.

   2. Enter the activation key that you got from the agent's local console.

7. Choose Create agent.

To create an Amazon EC2 agent

1. Deploy an Amazon EC2 agent.

2. Choose a service endpoint that the agent uses to communicate with AWS.

   In this situation, we recommend using a virtual private cloud (VPC) service endpoint.

3. Configure your network to work with VPC service endpoints.

4. Activate the agent.

1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

2. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

3. For Location type, choose Microsoft Azure Blob Storage.

4. For Agents, choose the DataSync agent that can connect with your Azure Blob Storage container.

   You can choose more than one agent. For more information, see Using multiple DataSync agents.

5. For Container URL, enter the URL of the container that's involved in your transfer.

6. (Optional) For Access tier when used as a destination, choose the access tier that you want your objects or files transferred into.

7. For Folder, enter path segments if you want to limit your transfer to a virtual directory in your container (for example, /my/images).

8. For SAS token, enter the SAS token that allows DataSync to access your blob storage.

   The token is part of the SAS URI string that comes after the storage resource URI and a question mark (?). A token looks something like this:

   sp=r&st=2023-12-20T14:54:52Z&se=2023-12-20T22:54:52Z&spr=https&sv=2021-06-08&sr=c&sig=aBBKDWQvyuVcTPH9EBp%2FXTI9E%2F%2Fmq171%2BZU178wcwqU%3D

9. (Optional) Enter values for the Key and Value fields to tag the location.

   Tags help you manage, filter, and search for your AWS resources. We recommend creating at least a name tag for your location.

10. Choose Create location.

1. Copy the following create-location-azure-blob command:

   aws datasync create-location-azure-blob \
     --container-url "https://path/to/container" \
     --authentication-type "SAS" \
     --sas-configuration '{
         "Token": "your-sas-token"
       }' \
     --agent-arns my-datasync-agent-arn \
     --subdirectory "/path/to/my/data" \
     --access-tier "access-tier-for-destination" \
     --tags [{"Key": "key1","Value": "value1"}]

2. For the --container-url parameter, specify the URL of the Azure Blob Storage container that's involved in your transfer.

3. For the --authentication-type parameter, specify SAS.

4. For the --sas-configuration parameter's Token option, specify the SAS token that allows DataSync to access your blob storage.

   The token is part of the SAS URI string that comes after the storage resource URI and a question mark (?). A token looks something like this:

   sp=r&st=2023-12-20T14:54:52Z&se=2023-12-20T22:54:52Z&spr=https&sv=2021-06-08&sr=c&sig=aBBKDWQvyuVcTPH9EBp%2FXTI9E%2F%2Fmq171%2BZU178wcwqU%3D

5. For the --agent-arns parameter, specify the Amazon Resource Name (ARN) of the DataSync agent that can connect to your container.

   Here's an example agent ARN: arn:aws:datasync:us-east-1:123456789012:agent/agent-01234567890aaabfb

   You can specify more than one agent. For more information, see Using multiple DataSync agents.

6. For the --subdirectory parameter, specify path segments if you want to limit your transfer to a virtual directory in your container (for example, /my/images).

7. (Optional) For the --access-tier parameter, specify the access tier (HOT, COOL, or ARCHIVE) that you want your objects or files transferred into.

   This parameter applies only when you're using this location as a transfer destination.

8. (Optional) For the --tags parameter, specify key-value pairs that can help you manage, filter, and search for your location.

   We recommend creating a name tag for your location.

9. Run the create-location-azure-blob command.

   If the command is successful, you get a response that shows you the ARN of the location that you created. For example:

   { 
       "LocationArn": "arn:aws:datasync:us-east-1:123456789012:location/loc-12345678abcdefgh" 
   }

1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

2. In the left navigation pane, expand Data transfer, then choose Locations.

3. Choose your Azure Blob Storage location.

   You can see details about your location, including any DataSync transfer tasks that are using it.

1. Copy the following describe-location-azure-blob command:

   aws datasync describe-location-azure-blob \
     --location-arn "your-azure-blob-location-arn"

2. For the --location-arn parameter, specify the ARN for the Azure Blob Storage location that you created (for example, arn:aws:datasync:us-east-1:123456789012:location/loc-12345678abcdefgh).

3. Run the describe-location-azure-blob command.

   You get a response that shows you details about your location. For example:

   {
       "LocationArn": "arn:aws:datasync:us-east-1:123456789012:location/loc-12345678abcdefgh",
       "LocationUri": "azure-blob://my-user.blob.core.windows.net/container-1",
       "AuthenticationType": "SAS",
       "Subdirectory": "/my/images",
       "AgentArns": ["arn:aws:datasync:us-east-1:123456789012:agent/agent-01234567890deadfb"],
   }

1. Copy the following update-location-azure-blob command:

   aws datasync update-location-azure-blob \
     --location-arn "your-azure-blob-location-arn" \
     --authentication-type "SAS" \
     --sas-configuration '{
         "Token": "your-sas-token"
       }' \
     --agent-arns my-datasync-agent-arn \
     --subdirectory "/path/to/my/data" \
     --access-tier "access-tier-for-destination"

2. For the --location-arn parameter, specify the ARN for the Azure Blob Storage location that you're updating (for example, arn:aws:datasync:us-east-1:123456789012:location/loc-12345678abcdefgh).

3. For the --authentication-type parameter, specify SAS.

4. For the --sas-configuration parameter's Token option, specify the SAS token that allows DataSync to access your blob storage.

   The token is part of the SAS URI string that comes after the storage resource URI and a question mark (?). A token looks something like this:

   sp=r&st=2022-12-20T14:54:52Z&se=2022-12-20T22:54:52Z&spr=https&sv=2021-06-08&sr=c&sig=qCBKDWQvyuVcTPH9EBp%2FXTI9E%2F%2Fmq171%2BZU178wcwqU%3D

5. For the --agent-arns parameter, specify the Amazon Resource Name (ARN) of the DataSync agent that you want to connect to your container.

   Here's an example agent ARN: arn:aws:datasync:us-east-1:123456789012:agent/agent-01234567890aaabfb

   You can specify more than one agent. For more information, see Using multiple DataSync agents.

6. For the --subdirectory parameter, specify path segments if you want to limit your transfer to a virtual directory in your container (for example, /my/images).

7. (Optional) For the --access-tier parameter, specify the access tier (HOT, COOL, or ARCHIVE) that you want your objects to be transferred into.

   This parameter applies only when you're using this location as a transfer destination.