TextAdvanced

Design an experiment that can actually conclude

Viktor Submitted by Viktor Added 23 days ago
I want to test {{change}} on {{surface}}. Baseline: {{metric and current value}}, traffic {{volume per week}}.

Design the test:
1. State the hypothesis as a mechanism, not a hope: "changing X will cause Y because Z".
2. Primary metric (one), guardrail metrics, and the metrics I should look at but not decide on.
3. Minimum detectable effect worth shipping — reason from the business value, not from convention. Then compute the required sample and duration at 80% and 90% power. Show the numbers.
4. Tell me honestly if my traffic can't detect a realistic effect in a reasonable time, and what to do instead (bigger swing, upstream metric, switchback, or don't test).
5. Randomisation unit and the interference risks with it.
6. Pre-register the decision rule: what result ships, what kills it, what's inconclusive — written before we look.
7. The three ways this test will be misread when the numbers come in.

No peeking advice, no 'run it and see'.

How to use it

Write step 6 down and share it before launch. Pre-registration is the only real defence against motivated reading.

Compatible popular AI tools

These tools are mapped to this prompt based on their capabilities.

People who liked this prompt

4 community likes