Define Hyperparameter Ranges

This guide shows how to use SageMaker APIs to define hyperparameter ranges. It also provides a list of hyperparameter scaling types that you can use.

Choosing hyperparameters and ranges significantly affects the performance of your tuning job. Hyperparameter tuning finds the best hyperparameter values for your model by searching over a range of values that you specify for each tunable hyperparameter. You can also specify up to 100 static hyperparameters that do not change over the course of the tuning job. You can use up to 100 hyperparameters in total (static + tunable). For guidance on choosing hyperparameters and ranges, see Best Practices for Hyperparameter Tuning. You can also use autotune to find optimal tuning job settings. For more information, see the following Autotune section.

Static hyperparameters

Use static hyperparameters for the following cases: For example, you can use AMT to tune your model using param1 (a tunable parameter) and param2 (a static parameter). If you do, then use a search space for param1 that lies between two values, and pass param2 as a static hyperparameter, as follows.

param1: ["range_min","range_max"]
param2: "static_value"

Static hyperparameters have the following structure:

"StaticHyperParameters": {
    "objective" : "reg:squarederror",
    "dropout_rate": "0.3"
}  

You can use the Amazon SageMaker API to specify key value pairs in the StaticHyperParameters field of the HyperParameterTrainingJobDefinition parameter that you pass to the CreateHyperParameterTuningJob operation.

Dynamic hyperparameters

You can use the SageMaker API to define hyperparameter ranges. Specify the names of hyperparameters and ranges of values in the ParameterRanges field of the HyperParameterTuningJobConfig parameter that you pass to the CreateHyperParameterTuningJob operation.

The ParameterRanges field has three subfields: categorical, integer, and continuous. You can define up to 30 total (categorical + integer + continuous) tunable hyperparameters to search over.

Dynamic hyperparameters have the following structure:

"ParameterRanges": {
    "CategoricalParameterRanges": [
        {
            "Name": "tree_method",
            "Values": ["auto", "exact", "approx", "hist"]
        }
    ],
    "ContinuousParameterRanges": [
        {
            "Name": "eta",
            "MaxValue" : "0.5",
            "MinValue": "0",
            "ScalingType": "Auto"
        }
    ],
    "IntegerParameterRanges": [
        {
            "Name": "max_depth",
            "MaxValue": "10",
            "MinValue": "1",
            "ScalingType": "Auto"
        }
    ]
}

If you create a tuning job with a Grid strategy, you can only specify categorical values. You don't need to provide the MaxNumberofTrainingJobs. This value is inferred from the total number of configurations that can be produced from your categorical parameters. If specified, the value of MaxNumberOfTrainingJobs should be equal to the total number of distinct categorical combinations possible.

Autotune

To save time and resources searching for hyperparameter ranges, resources or objective metrics, autotune can automatically guess optimal values for some hyperparameter fields. Use autotune to find optimal values for the following fields:

To use autotune, do the following:

AMT will determine if your hyperparameters and example values are eligible for autotune. Hyperparameters that can be used in autotune are automatically assigned to the appropriate parameter range type. Then, AMT uses ValueHint to select an optimal range for you. You can use the DescribeHyperParameterTrainingJob API to view these ranges.

The following example shows you how to configure a tuning job that uses autotune. In the configuration example, the hyperparameter max_depth has ValueHint containing an example value of 4.

config = {
    'Autotune': {'Mode': 'Enabled'},
    'HyperParameterTuningJobName':'my-autotune-job',
    'HyperParameterTuningJobConfig': {
        'HyperParameterTuningJobObjective': {'Type': 'Minimize', 'MetricName': 'validation:rmse'},
        'ResourceLimits': {'MaxNumberOfTrainingJobs': 5, 'MaxParallelTrainingJobs': 1},
        'ParameterRanges': {       
            'AutoParameters': [
                {'Name': 'max_depth', 'ValueHint': '4'}
            ]
        }
    },
    'TrainingJobDefinition': {
    .... }

Continuing the previous example, a tuning job is created after the previous configuration is included in a call to the CreateHyperParameterTuningJob API. Then, autotune converts the max_depth hyperparameter in AutoParameters to the hyperparameter IntegerParameterRanges. The following response from a DescribeHyperParameterTrainingJob API shows that the optimal IntegerParameterRanges for max_depth are between 2 and 8.

{
    'HyperParameterTuningJobName':'my_job',
    'HyperParameterTuningJobConfig': {
        'ParameterRanges': {
            'IntegerParameterRanges': [
                {'Name': 'max_depth', 'MinValue': '2', 'MaxValue': '8'},
            ],
        }
    },
    'TrainingJobDefinition': {
        ...
    },
    'Autotune': {'Mode': 'Enabled'}
    
}

Hyperparameter scaling types

For integer and continuous hyperparameter ranges, you can choose the scale that you want hyperparameter tuning to use. For example, to search the range of values, you can specify a value for the ScalingType field of the hyperparameter range. You can choose from the following hyperparameter scaling types:

For an example notebook that uses hyperparameter scaling, see these Amazon SageMaker AI hyperparameter examples on GitHub.