Experiment results
Once an experiment has run for a while, Personalization reports how each variant performed against the control. This page explains what the results tell you, what “statistically significant” means, and when it’s actually safe to declare a winner.
Reading the results
| Element | What it tells you |
|---|---|
| Control vs. variant | How each variant performed on your chosen success metric compared with the control, and against every other variant. |
| Sample size | How many visitors each control/variant has served — too few visitors makes any difference unreliable, no matter how large it looks. |
| Statistical significance | Whether the difference you see is likely real, not just random noise from a small or short-running sample. |
| Run time | How long the experiment has been live — significance needs both enough visitors and enough elapsed time to smooth out effects like day-of-week or time-of-day traffic patterns. |
Statistical significance, in plain terms
A variant that looks ahead of the control is not automatically a winner. With a small sample, random chance alone can make one version look better for a while. Statistical significance is the test that tells you whether an observed difference is “highly likely to be real, repeatable, and not the result of random chance” rather than a short-lived blip — the standard reasoning behind A/B-test significance testing in general.
Two practical consequences follow:
- Don’t call a winner early. The longer an experiment runs and the more visitors it serves, the more an apparent gap can be trusted. Stopping as soon as a variant “looks good” is a common way to call a false winner.
- No result (yet) is a valid result. If the experiment hasn’t reached enough visitors or run long enough, the honest reading is “not enough data yet” — not “no difference.”
If a result looks off
Once a variant is a confirmed, significant winner, promote it to 100% of traffic from the experiment setup. For how to configure the experiment itself — control, variants, traffic allocation, and success metric — see Create and manage experiments. For an aggregated view of pipeline and decision health feeding your experiments, see Personalization logs.
Discussion
Comments are not configured for local development. They appear once the site owner connects GitHub Discussions (Giscus).