A/B testing compares versions using deliberately assigned groups. It is used to estimate whether a change caused a difference in a chosen outcome.
A useful test begins with a hypothesis and an evaluation plan. Showing two designs and choosing the more popular one informally is a different activity.
What should be tested?
Choose a change linked to an observed problem or plausible explanation. A hypothesis should connect the evidence, proposed change and expected outcome.
Changing several unrelated elements can still test a combined experience, but it limits what can be attributed to any individual detail.
Why does assignment matter?
Groups should be comparable under the experimental design. If one version receives returning customers and another receives new visitors, their difference may reflect the audience.
Implementation also needs consistency. People unexpectedly moving between variants can complicate interpretation.
What measure should decide success?
Choose a primary outcome before reviewing results. Add guardrails where a local gain could create a wider problem, such as lower lead quality.
A higher button-click rate may be useful evidence, but it does not establish more completed enquiries or sales.
How much traffic is needed?
The requirement depends on the baseline, desired detectable effect and statistical method. There is no universal minimum that suits every website.
Low-volume businesses may find research and monitored improvements more practical than a conventional experiment. Be honest about the limits of causal claims in that case.
When should a test stop?
Define the evaluation approach in advance. Repeatedly stopping whenever an early result looks favourable can create misleading conclusions unless the method accounts for that approach.
Include the business cycle where relevant. A short window may capture an unusual day rather than typical behaviour.
How should results be reported?
State the sample, outcome definition, uncertainty and implementation conditions. Distinguish an inconclusive result from evidence of no meaningful effect.
Also explain the commercial interpretation. A measurable movement may be too small to justify operational complexity.
What happens after a winning result?
Check rollout and monitor the relevant outcomes. Implementation changes can affect whether the tested experience is preserved.
Record what was learned, including unsuccessful tests. A programme becomes more useful when the evidence informs future hypotheses rather than only producing a list of winners.
Further reading
For reference, see the supporting documentation. Explore related explanations of Conversion rate optimisation, Attribution, Prototype.
A practical example
Illustrative example: a business tests clearer consultation details against its current page. The primary outcome is completed requests, with suitability checked separately. It decides how to evaluate the test before looking at early differences.
Why it matters
A/B testing can support causal learning when the design and analysis are appropriate. Clear hypotheses and outcome definitions keep it focused on useful decisions.
