Experimentation and conversion rate optimisation
If you're comparing conversion rate optimisation agencies, ask how many visitors your next test needs. Here, every A/B test is sized before launch from your baseline rate and the smallest lift worth finding. Each readout ends in a decision: ship, don't ship or run longer.
A/B test readout, example
The variant lifted conversion by 10.4%, with an interval of +0.10 to +0.55 points
Check first, sample ratioPlanned split: 50/50Control 48,200 visitors, variant 48,150Chi-square p = 0.87: passed
| Arm | Visitors | Orders | Conversion |
|---|---|---|---|
| Control | 48,200 | 1,494 | 3.10% |
| Variant | 48,150 | 1,647 | 3.42% |
95% interval for the difference
Two-proportion normal approximation, unpooled standard error.- Relative lift
- +10.4%
- Decision
- Ship
- Difference
- +0.32 percentage points
- 95% interval for the difference
- +0.10 to +0.55 points
What this fixes
Four signs your tests aren't telling you much.
What a CRO consultant hands over
A plan before each test, and a log of every result.
The sample size is fixed before the test starts, and a good Tuesday doesn't change it.
How A/B testing runs, from plan to decision
From baseline rate to a decision, step by step.
- 1
Scoping
Baseline conversion comes from the order system, page by page, along with the weekly traffic each page gets. From those two numbers, work out the smallest lift a four-week test could detect. If that's 25%, you hear it now, and the roadmap tests bigger changes.
- 2
Build
Before launch, a test order goes through each arm in Safari on an iPhone and in Chrome on Android and desktop. Each order must show exactly one purchase event.
- 3
Run
Whole weeks only, so every weekday counts the same. The results stay closed until the planned sample is reached. The only reason to stop early is a variant that's broken.
- 4
Readout and handover
The readout is added to the test log beside its plan, in a file kept in your own drive. The next test is read against it.
Where A/B testing goes wrong
How tests mislead, and the guard written into each plan.
Every readout gives its interval and one decision. Ship, don't ship, or run longer, with the reason in a sentence.
Audits after a site migration
How to tell a real drop from broken tags.
A new site goes live, conversion drops, and the first question is whether it really dropped. So the audit starts with tracking. Purchase events are checked against the order system for the weeks before and after launch, and any tag that fires twice, or not at all, is found before a single rate is compared. Only then does it look at conversion by step, from product view to basket to checkout to order, split by device, against the same weeks a year earlier so the season doesn't mislead. The report names the step and the device where the drop sits. If tracking turns out to be the cause, that's tracking work.
Conversion rate optimisation by industry
What each test is read on, by sector.
A/B tests measure changes to the site. Whether the ads bringing visitors to it pay their way is a question for marketing measurement.
Questions people ask
Answers on traffic, inconclusive tests and migrations.
Do you run A/B testing for ecommerce sites?
Yes. Most ecommerce tests sit on the product page, the basket or the checkout, and the primary metric is orders per visitor taken from the order system, so a test can't win on clicks that never became sales.
Can you test a subscription paywall or sign-up flow?
Yes. A paywall test is read on paid conversions per reader who saw the paywall, and the winning variant is checked again at the first renewal, because a cheaper first month can win the test and lose the subscribers.
How much traffic do we need to run A/B tests?
It depends on your conversion rate and the smallest lift worth finding. At a 3.1% baseline, finding a 10.5% relative lift with 80% power at two-sided 5% significance takes about 47,000 visitors in each arm. Scoping works this out for your pages before any test is promised.
What if a test is inconclusive?
The readout says so, with the interval, and gives one of two next steps: run longer if the interval is close to a lift worth having, or stop and test a bigger change if it isn't.
Can you audit our conversion rate after a site migration?
Yes. Tracking gets checked against the order system first, then conversion is compared step by step and by device for the weeks before and after launch, and the report names the step and device where orders fell, or the tag that broke.
Book a call about your next test
Bring the page you want to test and its traffic. The first call works out what a test there could detect, and how long it would take.