Get recommendations for models with LoRA adapters

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains a small set of adapter weights on top of a frozen base model. You can serve many adapters that share a single base model on one endpoint. This is more cost-effective than deploying a separate copy of the model for each fine-tuned variant.

When you add adapters to a recommendation job, SageMaker AI sizes and benchmarks a multi-adapter deployment. This deployment is a single base model with one Inference components per adapter on the same endpoint. The recommendations account for the additional memory that the adapters require, so SageMaker AI validates the returned configurations against the base model and the adapters together.

Prerequisites

In addition to the Prerequisites for a recommendation job, you need the following to include adapters:

SageMaker AI reads your adapters with the IAM execution role that you pass in RoleArn. The role must be able to read (s3:GetObject and s3:ListBucket) the Amazon S3 location of each adapter. These are the same permissions that it uses for your base model artifacts. If you provide adapters as model packages, the role also needs sagemaker:DescribeModelPackage on each package so that SageMaker AI can resolve the Amazon S3 location of the adapter.

Requirements and restrictions

The following requirements and restrictions apply when you add adapters to a recommendation job:

Add adapters to a recommendation job

To include adapters, add the optional AdapterSource object to your CreateAIRecommendationJob request. AdapterSource is a union: set exactly one of the following fields, each a list of 1–10 adapter entries:

The AdapterId is a name that you assign to each adapter. It must meet the following requirements:

SageMaker AI uses the AdapterId as the inference component name for that adapter. This name is also the routing handle that you use to send requests to the adapter when you invoke the deployed endpoint.

Python (boto3) — adapters in Amazon S3

try:
    response = client.create_ai_recommendation_job(
        AIRecommendationJobName="my-lora-recommendation-job",
        ModelSource={
            "S3": {
                "S3Uri": "s3://amzn-s3-demo-bucket/models/my-base-model/",
            }
        },
        OutputConfig={
            "S3OutputLocation": "s3://amzn-s3-demo-bucket/recommendations/"
        },
        AdapterSource={
            "S3Uris": [
                {"AdapterId": "sql", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/sql/"},
                {"AdapterId": "chat", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/chat/"},
            ]
        },
        PerformanceTarget={
            "Constraints": [
                {"Metric": "throughput"}
            ]
        },
        AIWorkloadConfigIdentifier="my-recommendation-workload",
        RoleArn="arn:aws:iam::111122223333:role/ExampleRole",
    )
    print(response["AIRecommendationJobArn"])
except client.exceptions.ResourceInUse as error:
    print(f"A job with this name already exists: {error}")
except Exception as error:
    print(f"Error creating recommendation job: {error}")
    raise
        

Python (boto3) — adapters as model packages

try:
    response = client.create_ai_recommendation_job(
        AIRecommendationJobName="my-lora-recommendation-job",
        ModelSource={
            "S3": {
                "S3Uri": "s3://amzn-s3-demo-bucket/models/my-base-model/",
            }
        },
        OutputConfig={
            "S3OutputLocation": "s3://amzn-s3-demo-bucket/recommendations/"
        },
        AdapterSource={
            "ModelPackageArns": [
                {
                    "AdapterId": "sql",
                    "ModelPackageArn": "arn:aws:sagemaker:us-west-2:111122223333:model-package/my-sql-adapter/1",
                }
            ]
        },
        PerformanceTarget={
            "Constraints": [
                {"Metric": "throughput"}
            ]
        },
        AIWorkloadConfigIdentifier="my-recommendation-workload",
        RoleArn="arn:aws:iam::111122223333:role/ExampleRole",
    )
    print(response["AIRecommendationJobArn"])
except client.exceptions.ResourceInUse as error:
    print(f"A job with this name already exists: {error}")
except Exception as error:
    print(f"Error creating recommendation job: {error}")
    raise
        

AWS CLI

aws sagemaker create-ai-recommendation-job \
  --ai-recommendation-job-name "my-lora-recommendation-job" \
  --model-source '{"S3": {"S3Uri": "s3://amzn-s3-demo-bucket/models/my-base-model/"}}' \
  --output-config '{"S3OutputLocation": "s3://amzn-s3-demo-bucket/recommendations/"}' \
  --adapter-source '{
    "S3Uris": [
      {"AdapterId": "sql", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/sql/"},
      {"AdapterId": "chat", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/chat/"}
    ]
  }' \
  --performance-target '{"Constraints": [{"Metric": "throughput"}]}' \
  --ai-workload-config-identifier "my-recommendation-workload" \
  --role-arn "arn:aws:iam::111122223333:role/ExampleRole" \
  --region us-west-2
        

Interpret the recommendations

When the job completes, call DescribeAIRecommendationJob. In addition to the fields described in Step 3: Review recommendations, the response includes adapter-specific information.

The DeploymentConfiguration in each recommendation describes the multi-adapter, inference-component-based topology (the base model plus one inference component per adapter). ExpectedPerformance reports the benchmarked metrics for that configuration.

The following example shows the adapter-related fields of a DescribeAIRecommendationJob response for a job created with Amazon S3 adapters.

{
    "AIRecommendationJobName": "my-lora-recommendation-job",
    "AIRecommendationJobStatus": "Completed",
    "AdapterSource": {
        "S3Uris": [
            {"AdapterId": "sql", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/sql/"},
            {"AdapterId": "chat", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/chat/"}
        ]
    },
    "Recommendations": [
        {
            "AdapterDetails": {
                "S3Uris": [
                    {"AdapterId": "sql", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/sql/"},
                    {"AdapterId": "chat", "S3Uri": "s3://amzn-s3-demo-bucket/adapters/chat/"}
                ],
                "ModelPackageArns": [
                    {"AdapterId": "sql", "ModelPackageArn": "arn:aws:sagemaker:us-west-2:111122223333:model-package/my-lora-recommendation-job-adapters/1"},
                    {"AdapterId": "chat", "ModelPackageArn": "arn:aws:sagemaker:us-west-2:111122223333:model-package/my-lora-recommendation-job-adapters/2"}
                ]
            },
            "DeploymentConfiguration": {
                "InstanceType": "ml.g6e.12xlarge",
                "InstanceCount": 1,
                "CopyCountPerInstance": 2,
                "MinCpuMemoryRequiredInMb": 12288,
                "EnvironmentVariables": {
                    "SAGEMAKER_SHM_SIZE_MB": "2048"
                }
            },
            "ExpectedPerformance": [
                {"Metric": "throughput", "Stat": "average", "Value": "512.0", "Unit": "tokens/second"}
            ]
        }
    ]
}

To benchmark a deployed multi-adapter endpoint, or to compare adapters against one another, target the inference component for each adapter in your benchmark job. For more information, see Benchmark multi-LoRA endpoints.