Toolnestly
Home/All tools/A/B Test Significance Calculator
Free & unlimited

A/B Test Significance Calculator

Know if your test result is real or just noise, in one paste.

Control

Variant

Fill in visitors and conversions for both groups to see your result.

What statistical significance actually means

A variant converting higher than your control doesn't automatically mean it's actually better — it might just be random noise from a small sample. This calculator runs a two-proportion z-test, the standard method for comparing two conversion rates, and returns a p-value: the probability you'd see a difference this large (or larger) purely by chance if the two versions were actually identical.

A low p-value means the result is unlikely to be chance. It doesn't measure how big or important the difference is — that's what the uplift percentage is for.

How to use this tool

1Enter total visitors and conversions for your control (the original version).
2Enter the same for your variant (the version you're testing).
3Pick a confidence level — 95% is the standard default for most marketing tests.
4Check the verdict: significant means the difference is unlikely to be random chance at that confidence level.

Common mistakes this helps you avoid

  • Calling a winner after a few hundred visitors — small samples produce noisy, unstable results
  • Stopping a test the moment it first looks significant, which inflates false positives
  • Confusing a big-looking percentage difference with statistical confidence in that difference

FAQ

How many visitors do I need before checking this?
There's no universal number — it depends on your baseline conversion rate and the size of the effect you're trying to detect. As a rough floor, most tests need at least a few hundred conversions per variant before results stabilize.
What confidence level should I use?
95% is the standard default in marketing. Use 99% for high-stakes changes (pricing, checkout flow) where a false positive is costly, or 90% for low-risk, fast-iteration tests.
Why does it say 'not significant' even though the variant looks better?
A higher raw conversion rate doesn't guarantee statistical significance — with a small sample, that difference is still plausibly random noise. Let the test run longer.
Does this tool store the numbers I enter?
No. Every calculation runs in your browser — nothing is sent to or saved on our servers.

Related tools

ROAS Calculator→Ad Budget & Pacing Calculator→Email Subject Line Checker→