What is A/B testing?

For example, a website can compare its current inquiry form with a shorter version. Assign eligible visitors to the two versions during the same experiment, then measure completed inquiries. Google describes A/B tests as comparisons of variations, including entire pages or flows. A variant can contain several changes, although that makes their individual effects harder to identify.

Google: Website testing. Microsoft: Experiment design.

Define success before looking at results

A test turns a proposed change into a measurable question. A higher click rate may be unhelpful if completed purchases or page reliability deteriorate. Microsoft recommends defining a hypothesis and success metrics, alongside guardrail and data-quality metrics. This entry promises no typical conversion lift.

Microsoft: Experiment design.

Plan assignment, measurement and analysis

Specify who is eligible, the unit of randomization and how assignment remains consistent. Choose the outcome and the smallest effect worth detecting, then plan the sample using baseline performance, variability and statistical power. Run the comparison under the analysis method’s stopping rules. A calendar duration or fixed conversion count cannot establish reliability for every test.

Microsoft: Experiment design.

Illustrative comparison with no winner declared

Suppose a hypothetical inquiry-form test records 40 completed inquiries from 1,000 eligible assigned visitors in control A, and 50 from 1,000 in variant B. The observed rates are 4% and 5%, a difference of 1 percentage point or a 25% relative increase. This arithmetic describes the sample only; it does not establish statistical significance, a causal conclusion or future lift.

Hypothetical observed form results
VersionAssigned visitorsCompleted inquiriesObserved rate
A: current form1,000404%
B: shorter form1,000505%

A/B testing compares defined variants. Multivariate testing examines combinations of different elements and their interactions, such as button text and page imagery. Changing several elements together in one variant does not automatically make a test multivariate: the analysis may only estimate the bundled change. Incrementality testing asks a narrower question about the effect of exposure relative to a holdout.

Google: Website testing.

Google Ads: Conversion Lift.

Keep the website test temporary

Google recommends avoiding cloaking, using canonical annotations to the original when test variants have separate URLs, and using HTTP 302 for temporary test redirects. Remove test scripts and alternate URLs once the experiment ends. These are search implementation rules, not a statistical design or a promise of unchanged rankings.

Google: Website testing.

Check assignment and avoid casual early stopping

If the observed group sizes disagree materially with the intended assignment ratio, investigate before interpreting outcomes. Microsoft describes sample ratio mismatch as a warning that can invalidate an experiment. Monitor errors and serious harm during the test, but do not declare success merely because a repeatedly checked result first looks favorable. Early decisions need a method that accounts for repeated testing.

Microsoft: Experiment analysis.

Questions about A/B testing

How long should an A/B test run?

Plan duration and sample size for the outcome, eligible traffic and analysis method. Google says reliable-test duration varies with conversion rates and traffic; a universal two-week rule or fixed conversion count is insufficient.

Can an A/B variant change more than one element?

Yes. It can compare whole pages or flows, but the result may describe the combined change rather than the effect of each component. Multivariate testing explicitly examines multiple changes and their combinations.

Does a higher observed rate prove the variant won?

No. Check data quality, uncertainty, planned stopping rules and guardrail outcomes before making a decision. A sample difference alone does not establish a dependable improvement.

Conversion Rate · Incrementality Testing · Landing Page · Call to Action

Sources