Bring your existing Evaluators via API

Create Evaluator

Select API-based from the create menu to start building.

Configure endpoint

Add your API endpoint details including:

• Headers
• Query parameters
• Request body

For advanced transformations, use pre and post scripts under the Scripts tab.

Map response fields

Test your endpoint using the playground. On successful response, map your API response fields to:

• Score (required)
• Reasoning (optional)

This mapping allows you to keep your API structure unchanged.

Set pass criteria

Configure two types of pass criteria:

Pass query Define criteria for individual evaluation metrics

Example: Pass if clarity score > 0.8

Pass evaluator (%) Set threshold for overall evaluation across multiple entries

Example: Pass if 80% of entries meet the clarity criteria

Test your Evaluator

Test your Evaluator in the playground before using it in your workflows. The right panel shows input fields for all variables used in your Evaluator.

1. Fill in sample values for each variable
2. Click Run to see how your Evaluator performs
3. Iterate and improve your evaluator based on the results