A/B testing: complete guide

An A/B test without a hypothesis is an experiment without direction. The hypothesis should follow the structure: "If we change [variable], we expect [measurable outcome] because [data-based reason]." The reason is critical: without it, you learn nothing from the test, whether it wins or loses.

Hypotheses come from prior research: a heatmap showing nobody reaches the CTA, a recording where users cannot find the form, or an analytics insight revealing 70% drop-off at a checkout step. Without research, hypotheses are guesses.

Sample size determines how many visits you need before the result is statistically reliable. It depends on three factors: your current conversion rate, the minimum detectable effect (what improvement you want to detect) and the desired confidence level (usually 95%).

A sample size calculator (Optimizely, Evan Miller, AB Test Guide) tells you how many visits you need per variant. If the answer is 50,000 and your site gets 2,000 weekly visits, the test will take 25 weeks, which is likely not viable. Adjust the minimum detectable effect or find intermediate metrics with higher volume.

Statistical significance (usually at 95%) indicates there is less than a 5% probability that the observed result is due to chance. It does not mean the result is correct with 95% certainty: it means that if there were no real difference, you would see that result only 5% of the time.

The most serious error is stopping a test before reaching the required sample size because "it already looks like a winner." Results fluctuate enormously in the first hours and days. A prematurely stopped test has a false positive probability far higher than the nominal 5%.

Optimizely and VWO are the reference platforms for A/B testing with visual editors that let you create variants without code. Google Optimize was discontinued, but GA4 integrates basic tests and can be complemented with Optimizely or VWO.

For technical teams, tools like LaunchDarkly, Statsig or GrowthBook offer feature flags with integrated A/B testing, ideal for testing product changes beyond interface tweaks. The choice depends on your tech stack and the volume of tests you plan to run.

Stopping the test early is the most frequent and most dangerous mistake. The second is lacking a clear hypothesis: testing for the sake of testing generates shallow learnings and wastes valuable traffic.

Other mistakes include: changing multiple variables at once (you cannot tell what caused the result), ignoring seasonality (a test that starts on Black Friday and ends afterwards is unreliable) and not segmenting results (the overall winning variant may be the loser on mobile).

Multivariate testing (MVT) tests multiple combinations of changes simultaneously: for example, 3 headlines × 2 CTAs = 6 variants. It requires much more traffic but identifies the optimal combination of elements.

Multi-armed bandits allocate more traffic to the winning variant as data accumulates, maximising conversions during the test instead of waiting until the end. They are useful when the opportunity cost of showing the losing variant is high (high-volume ecommerce).