Experimentation and conversion rate optimisation

If you're comparing conversion rate optimisation agencies, ask how many visitors your next test needs. Here, every A/B test is sized before launch from your baseline rate and the smallest lift worth finding. Each readout ends in a decision: ship, don't ship or run longer.

Last reviewed

A/B test readout, example

The variant lifted conversion by 10.4%, with an interval of +0.10 to +0.55 points

Example data

Check first, sample ratioPlanned split: 50/50Control 48,200 visitors, variant 48,150Chi-square p = 0.87: passed

Result
ArmVisitorsOrdersConversion
Control48,2001,4943.10%
Variant48,1501,6473.42%
Fixed before launch: 47,000 visitors an arm, from a 3.1% baseline, a 10.5% smallest lift, 80% power at two-sided 5% significance, 28 days.

95% interval for the difference

Two-proportion normal approximation, unpooled standard error.
Relative lift
+10.4%
Decision
Ship
Difference
+0.32 percentage points
95% interval for the difference
+0.10 to +0.55 points
Example data: an invented checkout test. Both arms passed the 47,000 visitors planned before launch, the split is within chance of 50/50, and the whole interval sits above zero, so the decision is to ship.

What this fixes

Four signs your tests aren't telling you much.

We called it after three daysA test looked like a clear winner by Wednesday, went live, and the lift never showed up in the monthly numbers.
Every test winsThe testing tool declares a winner most months, and revenue hasn't moved in a year.
We don't have enough traffic to testNobody has worked out what a test on your traffic could detect, so nothing gets tested at all.
Conversion fell after the new site went liveNobody can say whether customers are buying less or the purchase tag stopped firing on the new checkout.

What a CRO consultant hands over

A plan before each test, and a log of every result.

The test planOne page per test, dated before launch. It names the hypothesis, the primary metric, the baseline rate, the smallest lift worth detecting, the sample size, the run length and the conditions for stopping early, which only cover a broken variant. The primary metric gets a written formula, the way each figure in our KPI library does.
The readoutThe sample ratio check first, then the result with its interval, then the decision in one line, laid out like the exhibit above.
The testing roadmapThe next tests in order, ranked by the traffic that reaches each page and the size of change it would take to show up.
The test logEvery test you've run, losers and inconclusive ones included, with its plan and readout, in a file you own.
The post-migration auditWhere a new site has gone live, a report on conversion by step and device before and after, described in its own section below.

The sample size is fixed before the test starts, and a good Tuesday doesn't change it.

How A/B testing runs, from plan to decision

From baseline rate to a decision, step by step.

  1. 1

    Scoping

    Baseline conversion comes from the order system, page by page, along with the weekly traffic each page gets. From those two numbers, work out the smallest lift a four-week test could detect. If that's 25%, you hear it now, and the roadmap tests bigger changes.

  2. 2

    Build

    Before launch, a test order goes through each arm in Safari on an iPhone and in Chrome on Android and desktop. Each order must show exactly one purchase event.

  3. 3

    Run

    Whole weeks only, so every weekday counts the same. The results stay closed until the planned sample is reached. The only reason to stop early is a variant that's broken.

  4. 4

    Readout and handover

    The readout is added to the test log beside its plan, in a file kept in your own drive. The next test is read against it.

Where A/B testing goes wrong

How tests mislead, and the guard written into each plan.

PeekingCheck a test every day and stop on the first day it looks significant, and you'll ship far more false winners than the 5% you think you're accepting. The guard is a sample size fixed before launch. If you need to look early, a sequential design gets agreed at the start, with its own stopping rule.
A broken splitThe split drifts from 50/50 because a redirect drops some visitors, a bot filter treats one arm differently, or the variant breaks the tracking. Any lift measured on that split is suspect. The guard is a sample ratio check before anything else. A failed check voids the readout, whatever the result says.
Twenty metrics, one winnerTrack enough metrics and one of them will look significant by chance. The guard is a single primary metric named in the plan. Everything else is reported as secondary and treated as a lead for the next test.

Every readout gives its interval and one decision. Ship, don't ship, or run longer, with the reason in a sentence.

Audits after a site migration

How to tell a real drop from broken tags.

A new site goes live, conversion drops, and the first question is whether it really dropped. So the audit starts with tracking. Purchase events are checked against the order system for the weeks before and after launch, and any tag that fires twice, or not at all, is found before a single rate is compared. Only then does it look at conversion by step, from product view to basket to checkout to order, split by device, against the same weeks a year earlier so the season doesn't mislead. The report names the step and the device where the drop sits. If tracking turns out to be the cause, that's tracking work.

Conversion rate optimisation by industry

What each test is read on, by sector.

EcommerceTests on the product page, basket and checkout, read on orders per visitor from the order system.
Subscription and mediaPaywall and sign-up tests, read at sign-up and again at the first renewal.

A/B tests measure changes to the site. Whether the ads bringing visitors to it pay their way is a question for marketing measurement.

Questions people ask

Answers on traffic, inconclusive tests and migrations.

Do you run A/B testing for ecommerce sites?

Yes. Most ecommerce tests sit on the product page, the basket or the checkout, and the primary metric is orders per visitor taken from the order system, so a test can't win on clicks that never became sales.

Can you test a subscription paywall or sign-up flow?

Yes. A paywall test is read on paid conversions per reader who saw the paywall, and the winning variant is checked again at the first renewal, because a cheaper first month can win the test and lose the subscribers.

How much traffic do we need to run A/B tests?

It depends on your conversion rate and the smallest lift worth finding. At a 3.1% baseline, finding a 10.5% relative lift with 80% power at two-sided 5% significance takes about 47,000 visitors in each arm. Scoping works this out for your pages before any test is promised.

What if a test is inconclusive?

The readout says so, with the interval, and gives one of two next steps: run longer if the interval is close to a lift worth having, or stop and test a bigger change if it isn't.

Can you audit our conversion rate after a site migration?

Yes. Tracking gets checked against the order system first, then conversion is compared step by step and by device for the weeks before and after launch, and the report names the step and device where orders fell, or the tag that broke.

Book a call about your next test

Bring the page you want to test and its traffic. The first call works out what a test there could detect, and how long it would take.

Book a call