From human annotation
We understand that having golden datasets when you start off is tough but this is a great need as you scale. In order to help you curate these datasets as you evolve your application, the platform allows for data curation from human annotations.
To curate dataset entries from your human annotations, follow the steps below:
In order to collect human annotations, you will have to set up a test run on a prompt or workflow and send the relevant results to human raters. Learn more about this process here
Once you have collected human ratings for a test run, go to the test run report
You will see a card for human evaluation as a part of the summary which shows the emails and status of each rater.
For raters who have completed the process, you can click on the View details button next to their email and open the detailed ratings view
You can check the ratings, comments as well as human-corrected output where applicable.
For cases that you want to add to a dataset, select the entries using the row checkboxes and click on the Add to dataset button at the top
You can now choose the relevant dataset and map the relevant data to the right columns in your dataset. Eg. If you are adding human-corrected output as ground truth within a golden dataset, you can select that column.
- For columns that you don't want to add to the dataset, simply uncheck the checkbox near its name