Creating a prompt dataset for Retrieve only evaluation jobs - Amazon Bedrock

Creating a prompt dataset for Retrieve only evaluation jobs

A Retrieve only evaluation jobs require a prompt dataset using JSON lines format. You can have up to 1000 prompts in your dataset

Key value pairs used in prompt dataset for Retrieve only evaluation job

  • referenceContexts – This parent key is used to specify the ground truth response you expect the Retrieve would return. Specify the ground truth in the text key. referenceContexts is required if you choose the Context coverage metric in your evaluation job.

  • prompt – This parent key is used to specify the prompt (user query) that you want the model to respond to while the evaluation job is running.

The following prompt is expanded for clarity. In your actual prompt dataset each line (a prompt) must be a valid JSON object.

{
    "conversationTurns": [{
        "referenceContexts": [{
            "content": [{
                "text": ""
            }]
        }],
        "prompt": {
            "content": [{
            "text": "This is a prompt"
            }]
        }
    }]
}