# Nitro Enclaves application development
An enclave application is an application that is designed and developed to run inside the isolated enclave environment. An enclave application typically consists of at least two components:
+ An application that runs on the parent instance
+ An application that runs inside the enclave
Due to the isolated environment of the enclave, the only channel of communication between applications that are running on the instance and applications that are running in the enclave is the vsock socket.
**Topics**
+ [Nitro Enclaves Developer AMI](#dev-ami)
+ [Nitro Enclaves SDK](#sdk)
+ [Application development on Linux](developing-applications-linux.md)
+ [Application development on Windows](developing-applications-windows.md)
## Nitro Enclaves Developer AMI
AWS provides a Nitro Enclaves Developer AMI that contains the tools and components needed to develop enclave applications and to build enclave image files. It also contains samples applications, such as hello-enclave, vsock\$1sample and kmstool, to demonstrate how to use and develop your own enclave applications. For more information, see [AWS Nitro Enclaves Developer AMI](https://aws.amazon.com//marketplace/pp/B08R69DKQ1).
## Nitro Enclaves SDK
The Nitro Enclaves SDK is a set of open-source libraries that you can use to develop your enclave applications. The SDKs also integrate with AWS KMS and provide built-in support for attestation and cryptographic operations. For more information about the SDKs and how to use them, see the [Nitro Enclaves SDK Github repository](https://github.com/aws/aws-nitro-enclaves-sdk-c).
# Nitro Enclaves application development on Linux instances
This section provides information for Nitro Enclaves application development on Linux instances.
## Getting started with the vsock: Vsock tutorial
**Important**
The vsock sample application is supported on Linux instances only.
The vsock sample application is a simple client-server application that exchanges information between the parent instance and the enclave using the vsock socket.
The vsock sample application includes a client application and a server application. The client application runs on the parent instance, while the server application runs in the enclave. The client application sends a simple text message over the vsock to the server application. The server application listens to the vsock and prints the message to the console.
The vsock sample application is available in both Rust and Python. This tutorial shows you how to set up and run the Rust vsock sample application. For more information about setting up and running the Python vsock sample application, see the [AWS Nitro Enclaves samples GitHub repository](https://github.com/aws/aws-nitro-enclaves-samples/tree/main/vsock_sample/py).
**Note**
The application source is also freely available from the [AWS Nitro Enclaves samples GitHub repository](https://github.com/aws/aws-nitro-enclaves-samples/tree/main/vsock_sample). You can use the application source as a reference for building your own applications.
**Prerequisites**
Ensure the following prerequisites are met to configure and test the sample application.
+ To test the sample application using the Nitro CLI, configure an enclave-enabled parent instance as detailed in [Step 1: Prepare the enclave-enabled parent instance](getting-started.md#launch-instance). This is the only step required from the getting started guide required to create the sample application.
+ To clone the sample repository from GitHub, install a Git client on the parent instance. For more information, see [Install Git](https://github.com/git-guides/install-git) in the GitHub documentation.
+ To compile the Rust vsock sample application, you must have Cargo, Rust’s build system and package manager installed. You must also add the `x86_64-unknown-linux-musl target` or `aarch64-unknown-linux-musl target`. To install and configure Rust, use the following commands.
```
$ curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh
```
**Important**
You must disconnect from the instance and then reconnect before running the following commands.
After you reconnect to the instance, determine the architecture of the operating system to configure and test the sample application. The architecture will be either `x86_64` or `aarch64`. Run the following command to display the architecture of the operating system.
```
echo $(uname -m)
```
Run the following commands to finish installing and configuring Rust.
```
$ rustup target add x86_64-unknown-linux-musl
```
```
$ sudo yum -y install gcc
```
**To try the vsock sample application**
1. Download the application source and navigate into the directory.
```
$ git clone https://github.com/aws/aws-nitro-enclaves-samples.git
```
```
$ cd aws-nitro-enclaves-samples/vsock_sample/rs
```
1. Compile the application code using Cargo.
```
$ cargo build --target=x86_64-unknown-linux-musl --release
```
The compiled binary is located in `vsock_sample/rs/target/x86_64-unknown-linux-musl/release/vsock-sample`.
1. Navigate two levels up.
```
$ cd ../..
```
1. Create a new file named `Dockerfile` and then add the following.
```
# start the Docker image from the Alpine Linux distribution
FROM alpine:latest
# copy the vsock-sample binary to the Docker file
COPY vsock_sample/rs/target/x86_64-unknown-linux-musl/release/vsock-sample .
# start the server application inside the enclave
CMD ./vsock-sample server --port 5005
```
1. Convert the Docker file to an enclave image file.
```
$ nitro-cli build-enclave --docker-dir ./ --docker-uri vsock-sample-server --output-file vsock_sample.eif
```
1. Boot the enclave using the enclave image file. You need to access the enclave console to see the server application output, so you must include the `--debug-mode` option.
```
$ nitro-cli run-enclave --eif-path vsock_sample.eif --cpu-count 2 --enclave-cid 6 --memory 256 --debug-mode
```
Make a note of the enclave ID, because you'll need this to connect to the enclave console.
1. Open the enclave console. The console provides a view of what's happening on the server side of the application.
```
$ nitro-cli console --enclave-id enclave_id
```
1. Open an SSH terminal window for the parent instance. You'll use this terminal to run the client application.
1. In the parent instance terminal, run the client application. When you start the client application, it automatically sends some text over the vsock to the server application running in the enclave. Watch the enclave console terminal for the output.
```
$ ./aws-nitro-enclaves-samples/vsock_sample/rs/target/x86_64-unknown-linux-musl/release/vsock-sample client --cid 6 --port 5005
```
1. When the server application receives the text over the vsock, it prints the text to the console.
```
$ [ 0.127079] Freeing unused kernel memory: 476K
[ 0.127631] nsm: loading out-of-tree module taints kernel.
[ 0.128055] nsm: module verification failed: signature and/or required key missing - tainting kernel
[ 0.154010] random: vsock-sample: uninitialized urandom read (16 bytes read)
Hello, world!
```
Now that you understand how the sample application works, download and customize the source code to suit your use case.
# Nitro Enclaves Application development on Windows instances
This section provides information for Nitro Enclaves application development on Windows instances.
**Topics**
+ [Considerations for using Nitro Enclaves on a Windows parent instance](#windows-considerations)
+ [Nitro Enclaves for Windows release notes](#release-notes)
+ [Subscribe to notifications of new versions](#sns-topic)
+ [Working with the vsock socket in Windows](vsock-win.md)
## Considerations for using Nitro Enclaves on a Windows parent instance
The EC2 parent instance and the enclaves operate as separate virtual machines. This means that each of them (the parent instance and all of its enclaves) must run its own operating system. The parent instance, supports both Linux and Windows (2016 and later) operating systems. However, the enclaves support only operating systems that support the Linux boot protocol. This means that even if you have a Windows parent instance, you must run a Linux environment inside your enclaves.
![\[Supported operating systems\]](http://docs.aws.amazon.com/enclaves/latest/user/images/enclave-os.png)
This also means that you must use a Linux-based instance to build your enclave image file (`.eif`).
**Topics**
Keep the following in mind when using a Windows parent instance.
+ Only Windows 2016 and later is supported on the parent instance.
+ You must run a Linux-based environment inside the enclave.
+ The Hello enclaves sample application is supported on Windows parent instances, but the enclave image file (`.eif`) must be built on a Linux instance. For more information, see [Getting started with the Hello Enclaves sample application](getting-started.md).
+ The KMS Tool sample application is supported on Windows parent instances, but the enclave image file (`.eif`) must be built on a Linux instance. For more information, see [Getting started with cryptographic attestation using the KMS Tool sample application](hello-kms.md).
+ On Windows, the vsock uses the standard Windows sockets (Winsock2) API. For more information, see [Working with the vsock socket in Windows](vsock-win.md).
+ AWS Certificate Manager for Nitro Enclaves is not supported with Windows parent instances.
+ To use the AWS Nitro Enclaves CLI software on your parent instance, you must install the **AWSNitroEnclavesWindows** package using AWS Systems Manager Distributor. For more information, see [Install the Nitro Enclaves CLI on Windows](nitro-enclave-cli-install-win.md).
+ The `nitro-cli build-enclave` command is not supported on Windows parent instances. For more information, see [nitro-cli build-enclave](cmd-nitro-build-enclave.md).
## Nitro Enclaves for Windows release notes
This section describes Nitro Enclaves (for Windows) features, improvements, and bug fixes by release date.
| Release date | version | Updates and bug fixes |
| --- | --- | --- |
| July 24, 2024 | 1.2.3 | The release updated the Nitro Enclaves for Windows installer to use WiX Toolset v5. |
| October 18, 2023 | 1.2.2 | The release improved installation of Nitro Enclaves for Windows and deprecated support for Windows Server 2012 R2. |
| March 27, 2023 | 1.2.1 | The release fixed a bug related to terminating multiple enclaves. This is the last version to support Windows Server 2012 R2. |
| May 4, 2022 | 1.2.0 | The release added the following commands, arguments, and output for Nitro CLI: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/enclaves/latest/user/developing-applications-windows.html) The release added the following bug fixes and enhancements: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/enclaves/latest/user/developing-applications-windows.html) |
| July 27, 2021 | 1.1.0 | The release added the following bug fixes and enhancements: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/enclaves/latest/user/developing-applications-windows.html) |
| April 27, 2021 | 1.0 | Initial release of Nitro Enclaves for Windows. |
## Subscribe to notifications of new versions
Amazon SNS can notify you when new versions of Nitro Enclaves for Windows are released. Use one of the following procedures to subscribe to these notifications.
------
#### [ Amazon SNS console ]
**To subscribe to notifications using the Amazon SNS console**
1. Open the Amazon SNS console at [https://console.aws.amazon.com/sns/v3/home](https://console.aws.amazon.com/sns/v3/home).
1. In the navigation bar, change the Region to **US West (Oregon)**, if necessary. You must select this Region because the SNS notifications that you are subscribing to are in this Region.
1. In the navigation pane, choose **Subscriptions**.
1. Choose **Create subscription**.
1. In the **Create subscription** dialog box, do the following:
1. For **Topic ARN**, enter `arn:aws:sns:us-west-2:404587003957:aws-nitro-enclaves-windows`.
1. For **Protocol**, choose `Email`.
1. For **Endpoint**, type an email address that you can use to receive the notifications.
1. Choose **Create subscription**.
1. You'll receive a confirmation email. Open the email and follow the directions to complete your subscription.
------
#### [ AWS Tools for PowerShell Core ]
**To subscribe to notifications using the Tools for Windows PowerShell**
Use the following command.
```
C:\> Connect-SNSNotification -TopicArn 'arn:aws:sns:us-west-2:404587003957:aws-nitro-enclaves-windows' -Protocol email -Region us-west-2 -Endpoint 'your_email_address'
```
------
#### [ AWS Command Line Interface ]
**To subscribe to notifications using the AWS CLI**
Use the following command.
```
C:\> aws sns subscribe \
--topic-arn arn:aws:sns:us-west-2:404587003957:aws-nitro-enclaves-windows \
--protocol email \
--notification-endpoint your_email_address
```
------
If you no longer want to receive these notifications, use the following procedure to unsubscribe.
**To unsubscribe to notifications using the Amazon SNS console**
1. Open the Amazon SNS console at [https://console.aws.amazon.com/sns/v3/home](https://console.aws.amazon.com/sns/v3/home).
1. In the navigation bar, change the Region to **US West (Oregon)**.
1. In the navigation pane, choose **Subscriptions**.
1. Select the check box for the subscription and then choose **Delete**. When prompted for confirmation, choose **Delete**.
# Working with the vsock socket in Windows
This topic provides information that is specific to working with the vsock socket on Windows instances.
**Topics**
+ [Terminology](#terminology)
+ [AWS vsock socket implementation](#implementation)
+ [Using the Winsock2 functions with vsock sockets](#apis)
+ [Unsupported Winsock2 functions](#unsupported-apis)
+ [Known issues](#known-issues)
## Terminology
**Service Provider Interface**
The Service Provider Interface (SPI) is a library registered with the Windows Sockets 2 (Winsock2) API to support a new address family for the vsock socket. The vsock SPI is available in a 64-bit version only. Only 64-bit applications can use vsock sockets.
**Port**
The port component of an address. This is a 32-bit unsigned value.
**Local address**
The address of a vsock socket on the host on which the application is running. The address includes a context identifier (CID) and a port. The CID and port value pairs are concatenated with a '`.`' when written as a string. For example, a host with CID of `3` that listens on port `1234` has a listening address of `3.1234`.
**Peer**
A host that this host is communicating with over the vsock socket.
**Remote address**
The address of the vsock socket of the peer. The address includes a context identifier (CID) and a port. The CID and port value pairs are concatenated with a '`.`' when written as a string. For example, a host with CID of `3` that listens on port `1234` has a listening address of `3.1234`.
## AWS vsock socket implementation
The following are considerations for vsock socket implementation using Winsock2.
**Topics**
+ [Build-time dependencies](#build-time)
+ [Runtime](#run-time)
+ [Loopback support](#loopback)
### Build-time dependencies
Some value definitions are required to create and interact with vsock sockets. These include the definitions of `AF_VSOCK`, `sockaddr_vm`, and some reserved values for CIDs and ports. It is recommended that you include these definitions by including the `VirtioVsock.h` header in your code. For more information, about the header, see the [ Nitro Enclaves SDK Github repository](https://github.com/aws/aws-nitro-enclaves-sdk-c/tree/main/include/aws/vsock/VirtioVsock.h).
### Runtime
To create a Winsock2 socket with the `AF_VSOCK` address family, the vsock SPI must be registered with Winsock2. The vsock SPI is registered during the AWS Nitro Enclaves installation. Currently, the vsock SPI is available in a 64-bit version and supports only 64-bit applications. For more information about installing AWS Nitro Enclaves on a Windows instance, see [Install Nitro CLI](nitro-enclave-cli-install-win.md#install-cli-win).
### Loopback support
Loopback is not supported with vsock sockets. Attempts to `connect()` to a CID that belongs to the same host could result in an error.
## Using the Winsock2 functions with vsock sockets
This section highlights differences between the Winsock2 functions and the AWS implementation for the vsock SPI.
**Note**
Functions not listed below follow the Winsock2 implementation and behave as described in the [Winsock2 API documentation](https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/) without any AWS specific nuances. For a list of the unsupported Winsock2 functions, see [Unsupported Winsock2 functions](#unsupported-apis).
**Topics**
+ [`WSAAccept()/accept()`](#WSAAccept)
+ [WSAAddressToString()](#WSAAddressToString)
+ [WSABind()/bind()](#WSABind)
+ [WSAConnect()/connect()](#WSAConnect)
+ [WSAEventSelect()](#WSAEventSelect)
+ [WSAGetPeerName()](#WSAGetPeerName)
+ [WSAGetSockName()](#WSAGetSockName)
+ [WSAGetSockOpt()/getsockopt()](#WSAGetSockOpt)
+ [WSAIoctl()/ioctlsocket()](#WSAIoctl)
+ [WSAListen()/listen()](#WSAListen)
+ [WSASend()/send()](#WSASend)
+ [WSASetSockOpt()/setsockopt()](#WSASetSockOpt)
+ [WSASocket()/socket()](#WSASocket)
+ [WSAStringToAddress()](#WSAStringToAddress)
+ [WSARecv()/recv()](#WSARecv)
### `WSAAccept()/accept()`
If a vsock transport reset or device disable event occurs after receiving a connection request, but before `accept()` is called, `accept()` returns an invalid socket and `WSAGetLastError()` returns the value `WSAECONNRESET`.
### WSAAddressToString()
Converts `sockaddr_vm` to a string in the `CID.Port` format.
### WSABind()/bind()
To create a connection using a specific local port, you must call `bind()` with a valid local CID and the desired local port before calling `connect()`. An enclave-enabled Amazon EC2 instance is always assigned local CID of `3`. A socket bound to `SOCKADDR_VM_CID_ANY` can only be used for listening and cannot be used with `connect()`. `SO_REUSEADDR` and `SO_EXCLUSIVEADDRUSE` are not configurable. AWS vsock sockets behave as if `SO_EXCLUSIVEADDRUSE` is enabled for all sockets. That is, if any socket is bound to a local `CID.port` pair, no other socket can bind to that same local `CID.port` except as an `accept()` from a listening socket that is bound to `SOCKADDR_VM_CID_ANY.port` or `CID.port`. Additionally, sockets cannot be bound to `SOCKADDR_VM_CID_ANY.port` when any other socket on the host is bound to an address with the same local port value and any CID.
### WSAConnect()/connect()
Outgoing connection requests have a non-configurable `1` second timeout before the peer responds with a connection acceptance packet. When using `WSAConnect()`, caller data and callee data are not supported. Specifying caller data results in an error and specifying callee data returns a length of `0`. QOS options are also not supported and return an error if specified.
### WSAEventSelect()
`FD_OOB`, `FD_QOS`, `FD_GROUP_QOS`, `FD_ROUTING_INTERFACE_CHANGE`, and `FD_ADDRESS_LIST_CHANGE` will never be signaled in the current implementation. However, they do not return an error if specified.
### WSAGetPeerName()
Gets the peer’s socket address as a `sockaddr_vm`.
### WSAGetSockName()
Gets the local socket address as a `sockaddr_vm`.
### WSAGetSockOpt()/getsockopt()
Only the following `optname` parameters are supported:
+ `SO_LINGER`
+ `SO_DONTLINGER`
+ `SO_RCVBUF`
+ `SO_ACCEPTCONN`
+ `SO_PROTOCOL_INFOW`
+ `SO_TYPE`
For more information, see [ getsockopt function](https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/nf-Winsock2-getsockopt) on the Windows app developer documentation website.
### WSAIoctl()/ioctlsocket()
Only the following `ioctls` are supported:
+ `FIONBIO`
+ `FIONREAD`
For more information, see [ ioctlsocket function](https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/nf-Winsock2-ioctlsocket) on the Windows app developer documentation website.
### WSAListen()/listen()
Setting a backlog size of `0` or less sets the backlog size to `0`. Setting the backlog size to a value greater than the implementation-specific maximum backlog size, which is currently `2048`, sets the backlog size to the implementation-specific maximum backlog size. Reducing the backlog size while connection requests exist on a listening socket rejects some of the connection requests until the number of connections is equal to, or less than, the new backlog size.
### WSASend()/send()
No flags are supported for these functions. The flag values must be set to `0`. If you specify a different value, an error is returned.
### WSASetSockOpt()/setsockopt()
This function follows the Winsock2 implementation. However, only the following options are supported:
+ `SO_LINGER`
+ `SO_DONTLINGER`
+ `SO_RCVBUF`
`SO_RCVBUF` has a minimum value of `4096` bytes and a maximum value of `2` MB. Requested buffer sizes are rounded down to a power of `2`, or to `2` MB if the value is greater than `2` MB. Received buffer size defaults to `256` KB.
For more information, see [ ioctlsocket function](https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/nf-Winsock2-ioctlsocket) on the Windows app developer documentation website.
### WSASocket()/socket()
This function returns a new `SOCKET`. With the AWS vsock SPI, this `SOCKET` is also a `HANDLE` that allows you to call functions, such as `ReadFile` and `WriteFile` directly on the `SOCKET`.
This function must be called with `af = AF_VSOCK` and `type = SOCK_STREAM`. Only the `WSA_FLAG_OVERLAPPED` flag is supported when calling `WSASocket()`, which allows overlapped IO on the `SOCKET` that is returned. If `socket()` is called, the `WSA_FLAG_OVERLAPPED` flag is set. For more information about overlapped creation of sockets, see [ socket function](https://docs.microsoft.com/en-us/windows/win32/api/Winsock2/nf-Winsock2-socket) on the Windows app developer documentation website.
### WSAStringToAddress()
Converts a string in the format of `CID.Port` to a `sockaddr_vm`.
### WSARecv()/recv()
Only the `MSG_PEEK` flag is supported for this function.
## Unsupported Winsock2 functions
The following Winsock2 functions are not supported with AWS vsock sockets.
+ `WSACancelBlockingCall()`
+ `WSAAsyncSelect()`
+ `WSAGetQosByName()`
+ `WSAJoinLeaf()`
+ `WSARecvDisconnect()`
+ `WSASendDisconnect()`
+ `WSARecvFrom()`
+ `WSASendTo()`
## Known issues
### Some IOs cannot be canceled
When calling `WSASend()`, `WSARecv()`, or `WSAIoctl()` on an overlapped socket, either with `lpOverlapped` omitted or with `lpCompletionRoutine` specified, the IO cannot be canceled by the user using `CancelIo` or `CancelIoEx`. `CancelIoEx` returns an error with `GetLastError()` returning `ERROR_NOT_FOUND`. All IOs can be canceled by calling `closesocket()`.