Why Would I Use Simulations Instead of Just Manual Testing?
Manual testing is time-consuming and often fails to uncover edge cases. Simulations automate this process, allowing you to:- Test Context Retention: Verify how your AI maintains context across multiple exchanges.
- Evaluate Emotional Intelligence: See how your agent responds to different user emotions (e.g., frustrated vs. happy) and behaviors.
- Verify Business Logic: Ensure the agent adheres to policies and uses business context correctly.
- Identify Dead-Ends: Spot conversation flows where the agent gets stuck or fails to resolve the user’s issue.
What Is the Difference Between Agent Simulation and Prompt Simulation?
Maxim offers two distinct simulation modes depending on what you are testing:| Simulation Type | Purpose | Best For |
|---|---|---|
| Agent Simulation | Tests complete agent workflows with AI-generated user interactions. | Multi-agent systems, complex workflows, and full application logic. |
| Prompt Simulation | Tests individual prompts or prompt chains with AI-generated inputs. | Optimizing prompt performance and logic before integration. |
How Do I Create a Realistic Test Scenario?
To ensure your simulation mirrors real-world usage, you must define three key components:The Scenario
Be specific about the situation you want to test. Vague scenarios lead to generic tests.
Example
“Customer requesting a refund for a defective laptop” or “New user needs help configuring account security settings.”
“Customer requesting a refund for a defective laptop” or “New user needs help configuring account security settings.”
The Agent Description
Provide a detailed description of your agent’s purpose, capabilities, and tone. This helps the simulation generator understand what the agent should be doing.
Example
“Financial Advisor Agent: Helps with investment decisions, analyzes portfolios, and maintains a professional tone. Always recommends consulting a licensed advisor for major decisions.”
“Financial Advisor Agent: Helps with investment decisions, analyzes portfolios, and maintains a professional tone. Always recommends consulting a licensed advisor for major decisions.”
Can I Control the Length and Complexity of the Simulation?
Yes. In the Advanced settings, you can configure constraints to tailor the simulation to your needs:- Maximum Number of Turns: Set a limit for the conversation length. If unset, the simulation runs until the conversation naturally concludes.
- Reference Tools: Attach specific Prompt Tools you want to test within the simulation (e.g., an API for checking order status).
- Reference Context: Add Context Sources (documents or knowledge bases) to guide the conversation.