Programmatic Evaluator

You can add code snippets which can then be used as an evaluator. For example the below shown function named validate takes three parameters: input, output, and expectedOutput. It aims to check if the output contains any special characters (such as !@#$%^&*(),.?":|<>). It uses a regular expression (specialCharRegex) to match any of these special characters against the output. If any special character is found in the output, the function returns true, indicating that the output is considered valid.

• Programmatic evaluators, akin to other evaluators , allow defining pass/fail criteria.
• They enable direct testing of functions.
  • For instance, if {run} is passed as output to the validate function, it returns true due to special characters { and }.
• Once created and tested, evaluators can be saved and seamlessly integrated into workflows or prompts.