Architecture overview

The same bucket holds a public leaderboard JSON object for each live race. Adding a track to an event writes an empty placeholder, the race functions rewrite it as rankings change, and CloudFront serves it to spectators on a short cache time to live (TTL) so standings are not shown stale.

The pre-signup trigger validates the username, applies the new-user compute and model limits from global settings, and creates the user’s profile.

The post-confirmation trigger adds the new user to the default racer group.

The custom message trigger records a metric each time the user pool sends an email.

When an administrator email is supplied at deployment, a custom resource creates that administrator and adds them to the admin group.

Profiles, models, evaluations, leaderboards, and submissions: create, read, update, and delete operations, and sending training and evaluation jobs to the job queue. Creating or retrying a model first checks the account’s SageMaker training quotas. If either is full, the model waits for capacity and no job is queued until the user retries. Only training jobs wait this way; an evaluation that SageMaker rejects for capacity fails instead, because there is no status it could recover from.

Model import and export: importing a virtual or physical model sends a job to the import queue. Retrieving a model’s asset URL starts the asset packaging function if no current package exists and reports the packaging as queued; a later call returns a pre-signed URL once the package is ready.

Live races: facilitator queue management, including listing the queue, reordering submissions via fractional indexing, removing submissions, resetting in-progress or failed models, and clearing the leaderboard. Launching a race starts the live race workflow. Other functions declare a winner, and grant newly authenticated users the IoT Core policy they need to subscribe to race topics.

Events: creating and editing events, their tracks and their fleets, moving an event through its statuses, recording runs and laps, and reading per-track and combined leaderboards. Deleting an event sends a cascade-delete request to the event delete queue.

Devices: activating cars and timers through Systems Manager hybrid activation, listing, updating, and deleting them individually or in batches, and sending restart, stop, color change, and clear models commands to them.

Model transfer: packaging a model for a physical car invokes the model optimizer, and deploying it to a car starts the push workflow.

Users and settings: creating a Cognito user for a racer registered at an event, bulk user creation, and reading and updating global settings. Bulk user creation runs a separate AWS Step Functions state machine that creates each submitted user and finalizes the batch once every entry has been processed.

The job dispatcher takes a job off the queue and starts the training workflow.

The job initializer sets up the job, creates its video stream, and starts the SageMaker training job.

The job monitor checks the status of the running job.

The job finalizer records the result, collects the job’s logs, and cleans up its video stream.

The live race workflow reuses the initializer, monitor, and finalizer to run race evaluations.

The DynamoDB stream triggers the stream handler. It starts a live race workflow execution when one or more submissions with PENDING status exist in the queue, the race is IN_PROGRESS, autolaunch is enabled, and no execution is currently running. It acquires the execution lock via a conditional write before starting the execution. Stream records it fails to process go to a dead-letter queue that raises an alarm.

The DynamoDB stream also triggers the broadcast handler, which detects relevant state changes such as evaluation started or completed, leaderboard updates, and winner declarations. It publishes them to IoT Core, writes the public leaderboard JSON object, and emits a race-submitted event to the race event bus when a run finishes. When a device record expires through DynamoDB TTL, it fans the expired instance ids out to the device pruner. Records it fails to process go to a dead-letter queue that raises an alarm as soon as any message arrives.

The SafetyNet function runs when a live race workflow execution reaches any terminal state. It clears the execution lock with a conditional write and applies a backoff check if the execution has failed repeatedly. It then touches a PENDING queue item to generate a DynamoDB stream event, which retriggers the stream handler if items remain in the queue.

The stats rebuild function recalculates aggregate race statistics in response to a race-submitted event, and is limited to one concurrent execution so that rebuilds are serialized.

Get next pending takes the first pending queue item by position and loads its submission.

Check autolaunch stops the loop when autolaunch is off or the race has completed.

Update queue status moves the item through in progress, completed, and failed, with a conditional check so a facilitator reset is not overwritten.

Clear execution lock releases the leaderboard’s lock as the last step, on both the success and the error path.

A custom event bus receives the race-submitted event from the broadcast handler and routes it to the stats rebuild function. Events that still fail once EventBridge has exhausted its retries go to a dead-letter queue, so a failed rebuild is retained for inspection rather than dropped.

A rule on the live race workflow’s execution status changes invokes the SafetyNet function whenever an execution succeeds, fails, aborts, or times out.

A rule captures Systems Manager instance association changes and command status changes, and invokes the state change handler.

A schedule invokes the device status poller every five minutes.

The state change handler records those changes against the matching device record, so the user interface reflects whether a car is online and how its last command finished.

The device status poller reads managed instance information from Systems Manager and refreshes the stored status of each registered device.

The device pruner deregisters the Systems Manager managed instance of each device whose record has expired through DynamoDB TTL.

The model optimizer, which runs from a container image in Amazon ECR, converts a trained or imported model into the format required by physical DeepRacer cars. Packaging a model invokes it asynchronously, and those requests go to a dead-letter queue when they fail, where a processor function marks the model’s optimization status as failed so that it does not stay stuck in progress. The import dispatcher invokes it synchronously instead, so a physical import that fails is retried by the import queue.

The push functions send the transfer command to the car, poll for its completion, and update the deployment status for the push workflow.

The import dispatcher takes a job off the import queue. For a virtual model, it starts the import workflow. For a physical model, it invokes the model optimizer directly.

The reward function validator, which runs from a container image in Amazon ECR, checks and sanitizes the customer-provided reward function code before it is saved to the system. The same function validates reward functions when a model is created or tested.

The model validator, which runs from a container image in Amazon ECR, checks the uploaded model.

Both validators run in a VPC of private isolated subnets with no NAT gateway, behind a security group that allows no outbound traffic, so customer-provided code cannot reach the network.

The import model assets function copies the model assets from the upload bucket into the user data bucket.

The import completion handler records the final status of the import, whether it succeeded or failed validation.

The failed request handler marks imports that reached the dead-letter queue as failed.