From logs
Logging helps you have visibility into your AI performance in production. The logs dashboards give you detailed tracing of each user interaction, performance metrics and the LLM output. The main aim of this process is to be able to learn from actual cases in production and use this vital feedback loop while iterating further. Data curation from logs helps bring real world use cases to the fore front and connect them into your evaluation datasets so you are constantly improving the quality of your system.
Curating data from logs
In order to see logs on your Maxim dashboard, you will have to integrate the Maxim SDK for your application. Learn more about logging here.
- As a part of the data that is logged you can also send user feedback score as a field for every entry.
Once you have your logs dashboard set up, you can view the table of all logs in real time on Maxim. Each trace will have information about the input, output, model, user feedback etc based on what you choose to send.
You can filter your logs based on any parameter that you send (Eg. User feedback == 0). This will provide you a list of the production cases that could be relevant to curate. Very good feedback could help in curating golden datasets and underperforming queries could help for further iterations.
Once you have the list of queries you want to add, select the rows using the checkboxes on the left edge of the row. Click the Add to dataset button in the table header.
You can choose the relevant dataset and then map the columns to dataset columns
- Datasets can have columns of type Input, Expected Output, etc. For columns that you don't want to add, simply uncheck the checkbox next to the column name. Learn more about dataset columns here