View a markdown version of this page

Train 3D Gaussian Splatting from your own video - Deadline Cloud
OverviewPrerequisitesSet up your farmCapture a video of a subjectSubmit the gsplat_pipeline jobMonitor the jobDownload and view the Gaussian Splatting .plyUnderstand the cost of your training jobsClean upNext stepsRelated resources

Train 3D Gaussian Splatting from your own video

This tutorial walks you through training your own 3D Gaussian Splatting point cloud from a video that you capture. You submit the Gaussian Splatting pipeline job bundle on GitHub to your Deadline Cloud farm. The job takes a video file as input and produces a Gaussian Splatting .ply file as output. When the job completes, you can view the result in any Gaussian Splatting viewer, such as SuperSplat.

The pipeline runs a single task that:

  1. Extracts video frames with the FFmpeg video processing tool.

  2. Solves Structure-from-Motion with COLMAP and GLOMAP, saving the pinhole model and undistorted images.

  3. Trains Gaussian Splatting with NeRF Studio splatfacto, Splatfacto in the Wild, or the simple_trainer.py gsplat library example, and saves the output to the .ply file you specify.

The following video demonstrates the Gaussian Splatting pipeline workflow on Deadline Cloud.

Estimated time: A few hours, including farm setup. Depending on the input video and the settings you select, the job itself can finish in 10 minutes or take hours.

Running this tutorial incurs charges for the GPU worker instances that process the job. The Understand the cost of your training jobs section shows how to review those costs.

Overview

To complete this tutorial, follow these steps:

  1. Complete the prerequisites.

  2. Set up your farm.

  3. Capture a video of a subject.

  4. Submit the gsplat_pipeline job.

  5. Monitor the job.

  6. Download and view the Gaussian Splatting .ply file.

  7. Clean up resources.

Prerequisites

Before you begin, complete the following setup:

  1. Create an AWS account if you do not already have one.

Set up your farm

You need a Deadline Cloud farm with a CUDA GPU fleet to run the Gaussian Splatting job. To set one up, complete the following steps:

  1. Follow the CUDA farm sample CloudFormation template instructions to create a Deadline Cloud farm that has a CUDA GPU fleet and can build conda packages.

  2. Follow the NeRF Studio sample conda package recipe instructions to build a NeRF Studio conda package into the Amazon S3 channel of your CUDA farm.

Note

If you only need the default NERFSTUDIO (splatfacto) trainer, you don't need to deploy the CUDA farm CloudFormation template or build the custom NeRF Studio conda package. The minimum requirements are a Deadline Cloud farm with a GPU fleet, a queue environment with conda-forge included in the conda channels, and the conda packages ffmpeg colmap glomap nerfstudio cuda. The GSPLAT_SIMPLE_TRAINER and NERFSTUDIO_SPLATFACTOW trainer options depend on commands that are only available in the custom-built NeRF Studio conda package.

Capture a video of a subject

You can use a video-capable camera like your smartphone to capture a video of a subject for your Gaussian Splatting. Consider the following tips:

  • Use a wide field of view, such as zoom level 0.5 in your camera app. With a wider field of view, more objects are common between image pairs for Structure-from-Motion to use.

  • Turn off video stabilization. Turning it off preserves identical lens optics between all frames, and can increase the quality of solves.

  • Plan your camera motion depending on the subject. To capture an object, like a bench or a bicycle, orbit around the subject a few times at different heights and distances. To capture a room interior, follow around the outside of the room with the camera facing inwards, and repeat at different camera heights. To capture less structured spaces such as outside terrain, think about how to include everything you want in your Gaussian Splatting, and how to capture all of it from multiple different angles.

  • Capture the video with slow and steady motion.

  • Keep moving the camera. Avoid stopping and panning from a single location.

Copy the video you captured from your camera to your computer for submitting to the farm.

Submit the gsplat_pipeline job

To submit the Gaussian Splatting job

  1. If you don't have a local copy of the deadline-cloud-samples GitHub repository, clone it or download it as a ZIP:

    git clone https://github.com/aws-deadline/deadline-cloud-samples.git
cd deadline-cloud-samples/job_bundles

  2. From the job_bundles directory, run the following command:

    deadline bundle gui-submit gsplat_pipeline

  3. Switch to the Job-specific settings tab and select paths for both the Input Video File and the Output Ply File.

  4. Choose Submit, or customize the settings first. If the input video has higher complexity, you may need to increase the Approximate Image Count value.

You can also select which Gaussian Splatting trainer to use and customize the CLI options for it. For example, to use the Monte Carlo Markov chain (MCMC) trainer with the bilateral grid option to produce up to 2 million splats, make the following changes:

  1. Switch the Gaussian Splatting Trainer from NERFSTUDIO to GSPLAT_SIMPLE_TRAINER.

  2. Modify the GSplat Simple Trainer Options text from --strategy.cap-max 1000000 to --strategy.cap-max 2000000 to get 2 million splats, and from --no-use-bilateral-grid to --use-bilateral-grid to enable the bilateral grid option.

Note

If you select an input video or options that require higher memory usage than the CUDA fleet provides, you need to update the fleet's minimum settings. Re-deploy your CloudFormation template with updated parameter values, or edit the settings from the Deadline Cloud console.

Monitor the job

After you submit the job, you can monitor its status from the Deadline Cloud monitor job table. Depending on the input video and the settings you selected, the job might finish in 10 minutes or take hours.

When the job is running, open the context menu for the task and choose View logs to open the log view. As the pipeline goes through its steps, it updates the status message visible in the task run details.

If you encounter errors, the log output in this view helps you track down the cause. Common causes include running out of memory or failing to solve for the camera poses. The pipeline can fail during any of these steps: FFmpeg frame extraction, GLOMAP Structure-from-Motion solving, or NeRF Studio training. To resolve errors, either adjust the fleet infrastructure or use a lower image resolution or image count.

Download and view the Gaussian Splatting .ply

To download and view the output

  1. When the job completes successfully, open the context menu for the completed task in the Tasks table of Deadline Cloud monitor and choose Download output. Depending on your settings, it either shows the download progress or presents a CLI command you can use to download. Deadline Cloud saves the .ply file to the location you selected when submitting the job.

  2. To view the result in your browser, open the SuperSplat Editor website and drag the file from your operating system's file browser onto the SuperSplat page.

  3. Toggle the Show/Hide Splats option, then rotate and move around the scene to find the subject of your capture.

With SuperSplat, you can also edit your Gaussian Splatting. For example, you can select a sphere, invert the selection, and delete the splats outside it to crop the scene. The SuperSplat documentation explains how to use the editor, or you can import your .ply file into your tool of choice.

Understand the cost of your training jobs

Deadline Cloud monitor includes a usage explorer feature that estimates the cost of the jobs you run, showing how much each training costs. If you're comfortable running your tests during off-peak hours when CUDA-capable instances are available with low enough interruption rates, a spot CUDA fleet costs less than an on-demand fleet. Usage explorer does not show costs outside of your job run time, such as idle worker instance time or storage on your Amazon S3 bucket.

Clean up

To avoid ongoing charges, clean up the resources that you created for this tutorial:

To clean up tutorial resources

  1. If you deployed the CUDA farm CloudFormation template, delete the CloudFormation stack from the CloudFormation console.

  2. If you used an existing farm and created a GPU fleet specifically for this tutorial, stop or delete that fleet. If you used a pre-existing shared fleet, leave it in place.

Next steps

You can pull apart, edit, and remix the gsplat_pipeline job bundle to suit your needs. You can customize your own 3D reconstruction pipeline, or follow the same patterns to run different CUDA workloads on your Deadline Cloud CUDA farm. The sample README works through the following directions in detail:

  • Run the job anywhere with the Open Job Description CLI, such as locally or on an Amazon EC2 instance, instead of on your farm.

  • Decompose the pipeline's single task into multiple Open Job Description steps connected by dependencies, following the pattern in the Job Development Progression sample on GitHub. Separate steps can run on different fleets, such as a CPU-only fleet for frame extraction and a GPU fleet for training.

  • Split the pipeline into multiple jobs with different structure. For example, one job to solve Structure-from-Motion once, and a second job to iteratively try different variations of Gaussian Splatting training parameters.

The following resources provide additional information: