Write the evaluation suite before an AI agent goes live
You are the evaluation lead for an AI agent that can use tools and affect real systems. Design the test suite before launch. Return: 1. BEHAVIOR CONTRACT - goals, non-goals, allowed tools, forbidden actions, autonomy boundary, and escalation conditions. 2. TEST SET - at least 20 cases across happy path, ambiguity, missing data, tool failure, conflicting instructions, stale state, prompt injection, permission limits, cost limits, and recovery. Table: Case | Input and setup | Expected actions | Forbidden actions | Pass rule | Severity. 3. GRADERS - deterministic checks where possible, rubric-based checks where needed, and required human review. 4. TRACE REQUIREMENTS - tool calls, decisions, costs, latency, state changes, and evidence needed to debug failures. 5. RELEASE GATE - pass thresholds by severity, regression rule, and who can approve exceptions. 6. MONITORING SAMPLES - production cases to review after launch. Rules: never pass a case solely because the final text looks good. AGENT PURPOSE: {{describe}} TOOLS AND PERMISSIONS: {{list}} POLICIES: {{paste}} FAILURE COSTS: {{describe}}
How to use it
Use sanitized real examples when possible. Keep critical safety cases deterministic and rerun the suite after prompt, model, or tool changes.
Compatible popular AI tools
These tools are mapped to this prompt based on their capabilities.
People who liked this prompt
1 community likes