Run a pipeline - Amazon SageMaker AI
Retrieve results from a pipeline run locallyRun a pipeline locally

Run a pipeline

The following page describes how to run a pipeline with Amazon SageMaker Pipelines, either with SageMaker AI resources or locally.

Start a new pipeline run with the pipeline.start() function as you would for a traditional SageMaker AI pipeline run. For information about the start() function, see sagemaker.workflow.pipeline.Pipeline.start.

Note

A step defined using the @step decorator runs as a training job. Therefore, be aware of the following limits:

  • Instance limits and training job limits in your accounts. Update your limits accordingly to avoid any throttling or resource limit issues.

  • The monetary costs associated with every run of a training step in the pipeline. For more details, refer to Amazon SageMaker AI Pricing.

Retrieve results from a pipeline run locally

To view the result of any step of a pipeline run, use execution.result(), as shown in the following snippet:

execution = pipeline.start()
execution.result(step_name="train")
Note

Pipelines does not support execution.result() in local mode.

You can only retrieve results for one step at a time. If the step name was generated by SageMaker AI, you can retrieve the step name by calling list_steps as follows:

execution.list_step()

Run a pipeline locally

You can run a pipeline with @step-decorated steps locally as you would for traditional pipeline steps. For details about local mode pipeline runs, see Run pipelines using local mode. To use local mode, provide a LocalPipelineSession instead of a SageMakerSession to your pipeline definition, as shown in the following example:

from sagemaker.workflow.function_step import step
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import LocalPipelineSession

@step
def train():
    training_data = s3.download(....)
    ...
    return trained_model
    
step_train_result = train()

local_pipeline_session = LocalPipelineSession()

local_pipeline = Pipeline(
    name="<pipeline-name>",
    steps=[step_train_result],
    sagemaker_session=local_pipeline_session # needed for local mode
)

local_pipeline.create(role_arn="role_arn")

# pipeline runs locally
execution = local_pipeline.start()