Use the fmeval library to run an automatic evaluation

Prerequisites

• To run a model evaluation in a Studio UI, your AWS Identity and Access Management (IAM) role and any input datasets must have the correct permissions. If you do not have a SageMaker Domain or IAM role, follow the steps in Guide to getting set up with Amazon SageMaker.

To set permissions for your Amazon S3 bucket

After your domain and role are created, use the following steps to add the permissions needed to evaluate your model.

1. Open the Amazon SageMaker console at https://console.aws.amazon.com/sagemaker/.

2. In the navigation pane, enter S3 into the search bar at the top of the page.

3. Choose S3 under Services.

4. Choose Buckets from the navigation pane.

5. In the General purpose buckets section, under Name, choose the name of the S3 bucket that you want to use to store your model input and output in the console. If you do not have an S3 bucket, do the following:

   1. Select Create bucket to open a new Create bucket page.

   2. In the General configuration section, under AWS Region, select the AWS region where your foundation model is located.

   3. Name your S3 bucket in the input box under Bucket name.

   4. Accept all of the default choices.

   5. Select Create bucket.

   6. In the General purpose buckets section, under Name, select the name of the S3 bucket that you created.

6. Choose the Permissions tab.

7. Scroll to the Cross-origin resource sharing (CORS) section at the bottom of the window. Choose Edit.

8. To add permissions to your bucket for foundation evaluations, ensure that the following code appears in the input box. You can also copy and paste the following into the input box.

   [
   {
       "AllowedHeaders": [
           "*"
       ],
       "AllowedMethods": [
           "GET",
           "PUT",
           "POST",
           "DELETE"
       ],
       "AllowedOrigins": [
           "*"
       ],
       "ExposeHeaders": [
           "Access-Control-Allow-Origin"
       ]
   }
   ]

9. Choose Save changes.

To add permissions to your IAM policy

1. In the search bar at the top of the page, enter IAM.

2. Under Services, select Identity and Access Management (IAM).

3. Choose Policies from the navigation pane.

4. Input AmazonSageMakerFullAccess into the search bar. Select the radio button next to the policy that appears. The Actions button can now be selected.

5. Choose the down arrow next to Actions. Two options appear.

6. Choose Attach.

7. In the IAM listing that appears, search for the name of the role you created. Select the check box next to the name.

8. Choose Attach policy.

Get started using Studio

1. In the search bar at the top of the page, enter SageMaker.

2. Under Services, select Amazon SageMaker.

3. Choose Studio from the navigation pane.

4. Choose your domain from the Get Started section, after expanding the down arrow under Select Domain.

5. Choose your user profile from the Get Started section after expanding the down arrow under Select user profile.

6. Choose Open Studio to open the landing page for Studio.

7. Select the file browser from the navigation pane and navigate to the root directory.

8. Select Create notebook.

9. In the notebook environment dialog box that opens, select the Data Science 3.0 image.

10. Choose Select.

11. Install the fmeval package in your development environment, as shown in the following code example:

    !pip install fmeval

    Note

    Install the fmeval library into an environment that uses Python 3.10. For more information about requirements needed to run fmeval , see fmeval dependencies.

Define a JumpStart model and configure a ModelRunner

1. Provide an endpoint by doing either of the following:

   • Specify the EndpointName to an existing JumpStart endpoint, the model_id, and model_version.

   • Specify the model_id and model_version for your model, and create a JumpStart endpoint.

   The following code example shows how create an endpoint for a Llama 2 foundation model that's available through JumpStart.

   import sagemaker
   from sagemaker.jumpstart.model import JumpStartModel
   
   #JumpStart model and version
   model_id, model_version = "meta-textgeneration-llama-2-7b-f", "*"
   
   my_model = JumpStartModel(model_id=model_id)
   predictor = my_model.deploy()
   endpoint_name = predictor.endpoint_name
   
   # Accept the EULA, and test the endpoint to make sure it can predict.
   predictor.predict({"inputs": [[{"role":"user", "content": "Hello how are you?"}]]}, custom_attributes='accept_eula=true')

   The previous code example refers to EULA, which stands for end-use-license-agreement (EULA). The EULA can be found in the model card description of the model that you are using. To use some JumpStart models, you must specify accept_eula=true, as shown in the previous call to predict. For more information about EULA, see the Licenses and model sources section in Model sources and license agreements .

   You can find a list of available JumpStart models at Built-in Algorithms with pre-trained Model Table.

2. Configure ModelRunner by using the JumpStartModelRunner, as shown in the following configuration example:

   from fmeval.model_runners.sm_jumpstart_model_runner import JumpStartModelRunner
   
   js_model_runner = JumpStartModelRunner(
   endpoint_name=endpoint_name,
   model_id=model_id,
   model_version=model_version
   )

   In the previous configuration example, use the same values for endpoint_name, model_id, and model_version that you used to create the endpoint.

3. Test your ModelRunner. Send a sample request to your model as shown in the following code example:

   js_model_runner.predict("What is the capital of London")

Define an Amazon Bedrock model and configure a ModelRunner

1. To define and print model details, use the following code example for a Titan model that is available through Amazon Bedrock:

   import boto3
   import json
   bedrock = boto3.client(service_name='bedrock')
   bedrock_runtime = boto3.client(service_name='bedrock-runtime')
   
   model_id = "amazon.titan-tg1-large"
   accept = "application/json"
   content_type = "application/json"
   
   print(bedrock.get_foundation_model(modelIdentifier=modelId).get('modelDetails'))

   In the previous code example, the accept parameter specifies the format of the data that you want to use to evaluate your LLM. The contentType specifies the format of the input data in the request. Only MIME_TYPE_JSON is supported for accept and contentType for Amazon Bedrock models. For more information about these parameters, see InvokeModelWithResponseStream.

2. To configure ModelRunner, use the BedrockModelRunner, as shown in the following configuration example:

   from fmeval.model_runners.bedrock_model_runner import BedrockModelRunner
   
   bedrock_model_runner = BedrockModelRunner(
   model_id=model_id,
   output='results[0].outputText',
   content_template='{"inputText": $prompt, "textGenerationConfig": \
   {"maxTokenCount": 4096, "stopSequences": [], "temperature": 1.0, "topP": 1.0}}',
   )

   Parametrize the ModelRunner configuration as follows.

   • Use the same values for model_id that you used to deploy the model.

   • Use output to specify the format of the generated json response. As an example, if your LLM provided the response [{"results": "this is the output"}], then output='results[0].outputText' returns this is the output.

   • Use content_template to specify how your LLM interacts with requests. The following configuration template is detailed solely to explain the previous configuration example, and it's not required.

     • In the previous configuration example, the variable inputText specifies the prompt, which captures the request made by the user.

     • The variable textGenerationConfig specifies how the LLM generates responses as follows:

       • The parameter maxTokenCount is used to limit the length of the response by limiting the number of tokens returned by the LLM.

       • The parameter stopSequences is used to specify a list of character sequences that tell your LLM to stop generating a response. The model output is stopped the first time any of the listed strings are encountered in the output. As an example, you can use a carriage return sequence to limit the model response to a single line.

       • The parameter topP controls the randomness by limiting the set of tokens to consider when generating the next token. This parameter accepts values between 0.0 and 1.0. Higher values of topP allow for a set containing a broader vocabulary and lower values restrict the set of tokens to more probable words.

       • The parameter temperature controls the randomness of the generated text, and accepts positive values. Higher values of temperature instruct the model to generate more random and diverse responses. Lower values generate responses that are more predictable. Typical ranges for temperature lie between 0.2 and 2.0.

       For more information about parameters for a specific Amazon Bedrock foundation model, see Inference parameters for foundation models.

     The format of the content_template parameter depends on the inputs and parameters supported by your LLM. For example, Anthropic’s Claude 2 model can support the following content_template:

     "content_template": "{\"prompt\": $prompt, \"max_tokens_to_sample\": 500}"

     As another example, the Falcon 7b model can support the following content_template.

     "content_template": "{\"inputs\": $prompt, \"parameters\":{\"max_new_tokens\": \
     10, \"top_p\": 0.9, \"temperature\": 0.8}}"

     Lastly, test your ModelRunner. Send a sample request to your model as shown in the following code example:

     bedrock_model_runner.predict("What is the capital of London?")