You can set up your annotation queue before adding evaluators, but you'll need at least one evaluator assigned before you can begin the annotation process.
Evaluating logs
Human annotation
Learn how to set up and use human annotations for evaluating your logs
Human annotations enable you to incorporate manual feedback into your log evaluation pipeline. This feature is crucial for validating model outputs, gathering training data, and maintaining quality control.
Setting up human evaluators
Navigate to your desired log repository and locate the Configure/Manage evaluation button in the top right corner.
From the dropdown menu, select Edit evaluation configuration to manage human evaluators.
In the configuration dialog, you'll find the Human evaluators section below Auto evaluation. Select the evaluators that are appropriate.
Using the annotation queue
You can use the annotation queue to annotate logs in 2 ways:
- Manual Selection: Hand-pick specific logs that need review
- Queue Logic: Set up rules to automatically add logs based on criteria
Manual Log Selection
When you need to review specific logs, manual selection provides direct control over your annotation queue:
In your log repository, click the Configure/Manage evaluation button in the top right corner.
Use the Add manually option to hand-pick logs for review.
Follow these steps to add logs for annotation:
- Click Add manually
- Select logs using the checkboxes
- Use the Add to annotation queue button that appears in the top right
Automated Queue Logic
For systematic collection of annotations, you can set up rules to automatically add logs to your queue:
Click the Configure/Manage evaluation button and select Create/View annotation queue.
In the configuration dialog, set up rules for automatically adding logs.
Annotating logs
Open your annotation queue by selecting View annotation queue from the evaluation menu.
Review your queued logs, which can come from both automatic rules ("Auto logic queue") and manual additions (showing the adding user's name).
Each log entry displays:
- Input and output content
- Added by (could be either the user or a rule)
- Rating options based on your evaluation setup
The scores shown are always the mean of all the scores provided by annotators.
To evaluate a log:
- Select the log for review
- Apply your rating (thumbs up/down or custom score)
- Add optional comments or re-write the output
- Save your evaluation by clicking Add rating
You can submit all your scores for a log at once by clicking on the log, adding your ratings and then clicking on the Save and next button.
You can also view the annotated scores that are submitted in the logs table, by scrolling towards the evaluator you wish to view the scores of.
