TextAdvanced

Decide the sample size before you run the test

Max Submitted by Max Added today
I want to test {{change}} on {{surface}}. Current baseline: {{metric and its current rate or mean, plus its variance if you have it}}. Traffic: {{visitors or events per day eligible for the test}}.

Work out, showing your reasoning in plain language:
1. The smallest effect that would actually change our decision — ask me for it if I haven't said, and explain why 'any improvement' is not an answer.
2. The sample size per arm for that effect at 80% and 90% power, and how many days each implies at my traffic.
3. Whether this test is feasible at all. If it needs more days than {{our patience}}, say so and give me the options: a bigger change, a proxy metric closer to the surface, or not testing.
4. The guardrail metrics that must not degrade, and their thresholds.
5. The exact stopping rule, plus what I should do when someone asks to peek on day three.

End with the one-paragraph test plan I can paste into the ticket.

How to use it

Answer question 1 honestly first. Most underpowered tests come from refusing to name the effect worth having.

Compatible popular AI tools

These tools are mapped to this prompt based on their capabilities.

People who liked this prompt

1 community likes