TextAdvanced

Write the eval before you ship the prompt

Viktor Submitted by Viktor Added 7 days ago
I'm shipping this prompt to production: {{paste prompt}}. It handles {{task}} at {{volume}}. A bad output causes {{consequence}}.

Build the eval:
1. Define what correct means for this task, as criteria that can be checked without me reading every output. Separate the objectively checkable from the ones needing judgement.
2. Write 15 test cases: 5 typical, 5 edge (empty, huge, multilingual, contradictory, adversarial), 3 that should be refused, and 2 where the right answer is "I don't know".
3. For each, the expected behaviour — not the exact expected text, unless format is the point.
4. Specify the automated checks: schema validity, required fields, forbidden content, length bounds, and any invariant I can assert in code.
5. For the judgement criteria, write the grading rubric a second model or a human would apply, with a scoring scale and examples of each score.
6. Define the regression bar: what pass rate ships, and which single failure blocks release regardless of the rate.
7. Name what this eval will not catch.

A prompt without an eval is a guess you can't defend.

How to use it

Keep the 15 cases in version control next to the prompt. Re-run them on every model upgrade — that's when things silently change.

Compatible popular AI tools

These tools are mapped to this prompt based on their capabilities.

People who liked this prompt

1 community likes