IVS Auto-Record to Amazon S3 | Low-Latency Streaming

This section provides information about the auto-record-to-S3 feature of Amazon IVS low-latency streaming. We discuss data storage for recorded Amazon IVS streams. We explain the storage contents and metadata file schema. We also discuss playback of your recorded content.

S3 Prefix

The S3 prefix is a unique directory structure for each live stream that is recorded. All media and metadata files for the live stream are written within this directory. For channels with recording enabled, the S3 prefix is generated when a live session starts and will be provided in the CloudWatch event at the start and end of a recording.

The S3 prefix has the following format:

/ivs/v1/<aws_account_id>/<channel_id>/<year>/<month>/<day>/<hours>/<minutes>/<recording_id>

Where:

For example:

ivs/v1/123456789012/AsXego4U6tnj/2020/6/23/20/12/j8Z9O91ndcVs

Recording Contents

When recording starts, video segments and metadata files are written to the S3 bucket that is configured for the channel. These contents are available for post-processing or playback as on-demand video.

Note that after a live stream starts and the Recording Start EventBridge event is emitted, it takes a little time before the manifest files and video segments are written. We recommend that you play back or process recorded streams only after the Recording End event is sent. (See Using Amazon EventBridge with IVS.)

The following is a sample directory structure and contents of a recording of a live Amazon IVS session:

ivs/v1/123456789012/AsXego4U6tnj/2020/6/23/20/12/j8Z9O91ndcVs/
   events
      recording-started.json
      recording-ended.json
   media
      hls
      thumbnails

The events folder contains the metadata files corresponding to the recording event. JSON metadata files are generated when recording starts, ends successfully, or ends with failures:

A given events folder will contain recording-started.json and either recording-ended.json or recording-failed.json.

These contain metadata related to the recorded session and its output formats. JSON details are given below.

The media folder contains all supported media contents, in two subfolders:

Important: The contents within the media folder are dynamically generated and determined by the characteristics of the first received video segments; the folder contents may not represent the ultimate characteristics (e.g., rendition quality). Do not make any assumptions about the static path. To discover the HLS renditions available and its path, use the JSON metadata files described below.

Byte-Range Playlists

The auto-record-to-S3 feature supports byte-range playlist generation, in addition to standard HLS playlists. Byte-range playlists conform to version 4 of the HLS specification. This allows for more fine-grained content clipping: in a byte-range playlist, each segment in a rendition index file references a subrange of bytes of a video chunk, providing more granularity than the standard 10-second media file size. With a byte-range playlist, the segment duration is the same as the keyframe interval configured for the stream.

Thumbnails

The thumbnailConfiguration property on a recording configuration allows you to enable or disable the recording of thumbnails for a live session and modify the interval at which thumbnails are generated for the live session. Thumbnail intervals may range from 1 second to 60 seconds; by default, thumbnail recording is enabled, at an interval of 60 seconds. For details, see the Amazon IVS API Reference.

Thumbnail configuration also may include the storage field (SEQUENTIAL and/or LATEST) and a resolution (LOWEST_RESOLUTION, SD, HD, or FULL_HD). Below are the resolutions for each option:

160 <= LOWEST_RESOLUTION <= 360

360 < SD <= 480

480 < HD <= 720

720 < FULL_HD <= 1080

Merge Fragmented Streams

The recordingReconnectWindowSeconds property on a recording configuration allows you to specify a window of time (in seconds) during which, if your stream is interrupted and a new stream is started, Amazon IVS tries to record to the same S3 prefix as the previous stream. In other words, if a broadcast disconnects and then reconnects within the specified interval, the multiple streams are considered a single broadcast and merged together.

IVS Recording State Change events in Amazon EventBridge: Recording End events and recording-ended JSON metadata files are delayed by at least recordingReconnectWindowSeconds, as Amazon IVS waits to ensure a new stream is not started.

For instructions on setting up the merge-streams functionality, see Step 4: Create a Channel with Optional Recording in Getting Started with Amazon IVS.

Eligibility

For multiple streams to record to the same S3 prefix, certain conditions must be met for all the streams:

Notes:

Known Issue

If recordingReconnectWindowSeconds is enabled and the Web Broadcast SDK is used, recording to the same S3 prefix may not work, as the Web Broadcast SDK dynamically changes bitrates and qualities.

JSON Metadata Files

When a recording state-change event occurs, a corresponding Amazon CloudWatch metric is generated and a metadata file is written within the S3 prefix. (See Monitoring Amazon IVS Low-Latency Streaming.)

This metadata is in JSON format. It comprises the following information.

Example: recording_started.json

{
   "version": "v1",
   "channel_arn": "arn:aws:ivs:us-west-2:123456789012:channel/AsXego4U6tnj",
   "recording_started_at": "2020-06-12T12:53:26Z",
   "recording_status : "RECORDING_STARTED",
   "media": {
      "hls": {
         "path": "media/hls",
         "playlist": "master.m3u8",
         "byte_range_playlist": "byte-range-multivariant.m3u8",
         "renditions": [
            {
               "path": "480p30",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 480,
               "resolution_width": 852
            },
            {
               "path": "360p30",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 360,
               "resolution_width": 640
            },
            {
               "path": "160p30",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 160,
               "resolution_width": 284
            },
            {
               "path": "720p60",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 720,
               "resolution_width": 1280
            }
         ]
      },
      "thumbnails": {
         "path": "media/thumbnails",
         "resolution_height": 480,
         "resolution_width": 852
      },
      "latest_thumbnail": {
         "path": "media/latest_thumbnail/thumb.jpg",
         "resolution_height": 480,
         "resolution_width": 852
      }
   }
}

Example: recording_ended.json

{
   "version": "v1",
   "channel_arn": "arn:aws:ivs:us-west-2:123456789012:channel/AsXego4U6tnj",
   "recording_ended_at": "2020-06-14T12:53:20Z",
   "recording_started_at": "2020-06-12T12:53:26Z",
   "recording_status": "RECORDING_ENDED",
   "media": {
      "hls": {
         "duration_ms": 172794489,
         "path": "media/hls",
         "playlist": "master.m3u8",
         "byte_range_playlist": "byte-range-multivariant.m3u8",
         "renditions": [
            {
               "path": "480p30",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 480,
               "resolution_width": 852
            },
            {
               "path": "360p30",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 360,
               "resolution_width": 640
            },
            {
               "path": "160p30",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 160,
               "resolution_width": 284
            },
            {
               "path": "720p60",
               "playlist": "playlist.m3u8",
               "byte_range_playlist": "byte-range-variant.m3u8",
               "resolution_height": 720,
               "resolution_width": 1280
            }
         ]
      },
      "thumbnails": {
         "path": "media/thumbnails",
         "resolution_height": 480,
         "resolution_width": 852
      },
      "latest_thumbnail": {
         "path": "media/latest_thumbnail/thumb.jpg",
         "resolution_height": 480,
         "resolution_width": 852
      }
   }
}

Example: recording_failed.json

{
   "version": "v1",
   "channel_arn": "arn:aws:ivs:us-west-2:123456789012:channel/AsXego4U6tnj",
   "recording_ended_at": "2020-06-14T12:53:20Z",
   "recording_started_at": "2020-06-12T12:53:26Z",
   "recording_status": "RECORDING_ENDED_WITH_FAILURE",
   "recording_status_message": "InternalServerException",
   "media": {
      "hls": {
         "duration_ms": 172794489,
         "path": "media/hls",
         "playlist": "master.m3u8",
         "renditions": [
            {
               "path": "480p30",
               "playlist": "playlist.m3u8",
               "resolution_height": 480,
               "resolution_width": 852
            },
            {
               "path": "720p60",
               "playlist": "playlist.m3u8",
               "resolution_height": 720,
               "resolution_width": 1280
            }
         ]
      },
      "thumbnails": {
         "path": "media/thumbnails",
         "resolution_height": 480,
         "resolution_width": 852
      },
      "latest_thumbnail": {
         "path": "media/latest_thumbnail/thumb.jpg",
         "resolution_height": 480,
         "resolution_width": 852
      }
   }
}

Discovering the Renditions of a Recording

When you stream content to an Amazon IVS channel, auto-record-to-s3 uses the source video to generate multiple renditions. Using Adaptive Bitrate Streaming (ABR), the Amazon IVS Player automatically switches the renditions (bitrates) as needed to optimize playback for varying network conditions.

Each rendition generated during live streaming is recorded in a unique path within the S3 recording prefix. The resolution detail, path, and playlist file names are stored in a JSON metadata file during the start and stop of the recording. If the recording configuration’s renditionSelection value is ALL, all renditions are selected for recording. If renditionSelection is CUSTOM, the user must select one or more of the following options: LOWEST_RESOLUTION, SD, HD, and FULL_HD. Below are the resolutions for each option:

160 <= LOWEST_RESOLUTION <= 360

360 < SD <= 480

480 < HD <= 720

720 < FULL_HD <= 1080

Important: Do not make any assumptions about the static rendition path or the list of generated renditions, as these are subject to change. Do not assume that a specific rendition will always be available for an Amazon IVS recording. To determine the available renditions, resolutions, and paths, refer to the metadata files.

The event/recording_started.json or event/recording_ended.json file within the recording prefix contains the paths and names of media files within the recording prefix. All path elements are relative to the previous path in the hierarchy. Elements under media > hls describe HLS assets, with master playlist name and path defined at this level.

Here is a Python code snippet that shows how to generate a master playlist path using the S3 recording prefix and metadata file:

def get_master_playlist(metadata_json, s3_recording_prefix):
   return s3_recording_prefix + '/' + metadata_json['media']['hls']['path'] + '/' + metadata_json['media']['hls']['playlist']

Elements under media > hls > renditions describe the list of renditions recorded. The resolution_height and resolution_width properties can be used to identify the video resolution. The path and playlist elements can be used to derive the rendition playlist path. Use these fields to determine which rendition to use for any post processing.

To discover the highest available rendition playlist for a recording, you can subscribe to "IVS Recording State Change" EventBridge events. (See Using Amazon EventBridge with IVS.) Below is a sample Python script that illustrates using a lambda function subscribed to those events.

import json
import boto3
s3 = boto3.resource('s3')

def get_highest_rendition_playlist(bucket_name, prefix_name):
   object_path = "{}/events/recording-started.json".format(prefix_name)
   object = s3.Object(bucket_name, object_path)
   body = str(object.get()['Body'].read().decode('utf-8'))
   metadata = json.loads(body)
   media_path = metadata["media"]["hls"]["path"]
   renditions = metadata["media"]["hls"]["renditions"]

   highest_rendition = None
   highest_rendition_size = 0

   for rendition in renditions:
       current_rendition_size = rendition["resolution_height"]
       if (current_rendition_size > highest_rendition_size):
           highest_rendition_size = current_rendition_size
           highest_rendition = rendition

   highest_rendition_playlist = media_path + '/' + highest_rendition['path'] + '/' + highest_rendition['playlist']
   return highest_rendition_playlist


def lambda_handler(event, context):
   prefix_name = event["detail"]["recording_s3_key_prefix"]
   bucket_name = event["detail"]["recording_s3_bucket_name"]
   rendition_playlist = get_highest_rendition_playlist(bucket_name, prefix_name)
   print("Highest rendition playlist: {}/{}".format(prefix_name, rendition_playlist))

   return {
       'statusCode': 200,
       'body': rendition_playlist
   }

Playback of Recorded Content from Private Buckets

Objects recorded with the Auto-Record to Amazon S3 feature are private by default; hence, these objects are inaccessible for playback using the direct S3 URL. If you try to open the HLS master manifest (m3u8 file) for playback using the Amazon IVS player or another player, you will get an error (e.g., "You do not have permission to access the requested resource"). Instead, you can play back these files with the Amazon CloudFront CDN (Content Delivery Network).

Amazon CloudFront Distribution

CloudFront distributions can be configured to serve content from private buckets. Typically this is preferable to having openly accessible buckets where reads bypass the controls offered by CloudFront. Your distribution can be set up to service from a private bucket by creating an origin access control (OAC), which is a special CloudFront user that has read permissions on the private origin bucket. You can create the OAC after you create your distribution, through the CloudFront console or API. See Creating a new origin access control.

Playback from Amazon CloudFront

Once you have set up your distribution using an OAC to gain access to your private bucket, your video files should be available for consumption through the CloudFront URL. Your CloudFront URL is the Distribution domain name on the Details tab in the AWS CloudFront console. It should be something like this:

a1b23cdef4ghij.cloudfront.net.

To stream your recorded video through your distribution, find the object key for your master.m3u8 file. It should be something like this:

ivs/v1/012345678912/a0bCDeFGH1IjK/2021/4/20/12/03/aBcdEFghIjkL/media/hls/master.m3u8

Append the object key to the end of your CloudFront URL. Your final URL will be something like this:

https://a1b23cdef4ghij.cloudfront.net/ivs/v1/012345678912/a0bCDeFGH1IjK/2021/4/20/12/03/aBcdEFghIjkL/media/hls/master.m3u8

To play back from a web browser, make sure to configure CORS in both CloudFront and S3 bucket. For CloudFront configuration, follow the instructions in Creating origin request policies to attach a CORS-S3 Origin request policy and SimpleCORS response header policy to the CloudFront distribution. See the example configuration console page below:

For S3 CORS configuration, see CORS configuration to create appropriate rules for your S3 bucket.

Now you can play back your recorded video as if you were playing directly from a bucket.

For more information, see Restricting access to an Amazon S3 origin.