A/B testing is a controlled experiment that compares two versions of a webpage, email, or ad to determine which one produces more conversions. Traffic is split randomly between a control (A) and a variant (B), and the winner is chosen only once the difference between them reaches statistical significance — usually at 95% confidence.
A/B testing replaces opinion with data. Instead of debating whether a green button or a blue one converts better, you run the test and let the numbers decide. It is the operating system of every serious CRO, growth, and product team.
What is A/B testing?
A/B testing is the practice of showing two versions of the same asset to separate visitor segments, then measuring which one produces a better outcome — a click, signup, purchase, or any other conversion event. Marketers, product teams, and growth operators run A/B tests on everything from email subject lines to entire checkout flows.
The mechanics are borrowed from clinical trials. A random slice of traffic sees version A (the current design, called the control). Another random slice sees version B (the change, called the variant). Because assignment is random, the only meaningful difference between the two groups is the change you made — so any gap in the conversion rate can be attributed to that change.
Companies that run A/B tests on landing pages see an average conversion rate lift of 12–15%, according to VWO's benchmark data. Small changes, measured properly, compound fast.
Why A/B testing matters
Getting your conversion rate right can double revenue without spending a dollar more on traffic. A/B testing is how you get there. Four reasons every growth team runs it:
- Reduces wasted ad spend. A 1% CRO lift on a $50K/month ad budget equals $6,000+ in recovered revenue annually — with no increase in media cost.
- Kills internal arguments with data. No more "I think the headline should say X." Run the test, ship the winner, and move on to the next hypothesis.
- Compounds user experience improvements. Each winning test stacks on the last. Ten 5% wins in a year is a 63% compound lift.
- Works across every channel. Email, paid ads, product pages, CTAs, pricing pages, onboarding flows — if users interact with it, you can test it.
How A/B testing works
Every A/B test follows the same four-step loop. Skip a step and the result is either wrong or unusable.
1. Hypothesis → "Shortening the form will lift signups by 10%"
2. Split → 50% see A (control), 50% see B (variant)
3. Measure → Run until 1,000+ conversions per variant
4. Decide → Ship winner, log learning, next hypothesis
Pick one variable
Choose a single element to test — a headline, button color, image, or offer. Testing multiple changes at once is called multivariate testing and needs far more traffic to interpret cleanly. Keep A/B tests to one variable.
Split your traffic
Your testing tool randomly sends 50% of visitors to version A and 50% to version B. Both groups should be comparable in size and source. Never split by day of week or by device — that introduces confounding variables.
Run until statistical significance
Do not call a winner after 48 hours and 200 visitors. Most tests need 1,000–5,000 conversions per variation to hit 95% confidence. Ending a test too early is the single most common A/B testing mistake and produces false positives roughly 30% of the time.
Analyze and ship
If the variant wins, implement it permanently. If it loses, document what you learned. Either outcome is useful — the only waste is not testing at all.
Real A/B testing examples
| Test | Control (A) | Variant (B) | Result |
|---|---|---|---|
| SaaS trial signup CTA | "Start Free Trial" | "Try It Free. No Credit Card." | +14% signups |
| Plumber email subject line | "Schedule your annual maintenance" | "Your furnace checkup is overdue" | +22% open rate |
| Ecommerce PDP hero image | Studio product shot | In-context lifestyle shot | +9% add-to-cart |
| Pricing page toggle | Monthly default | Annual default (with monthly toggle) | +18% ACV |
A/B testing vs multivariate testing — which to use
Both are experimentation frameworks, but they answer different questions and need very different traffic volumes.
Use A/B testing when
- You have a single, clear hypothesis
- Traffic is modest (under 100K sessions/month)
- You want a fast, clean answer
- The change is high-stakes (checkout, pricing)
- You are new to experimentation
Use multivariate when
- You have a fully redesigned page to tune
- Traffic is high (500K+ sessions/month)
- You want to isolate interaction effects
- You need to test 3+ elements at once
- Your CRO team is experienced
7 A/B testing best practices
- Write a hypothesis before the test. "If we [change X], then [metric Y] will improve by [Z%], because [reason]." Every winning team runs this format.
- Test one variable at a time. Anything else is multivariate — and needs 5–10x the traffic to interpret.
- Reach 95% confidence before deciding. Anything below is guessing. Use a sample size calculator up front.
- Run through at least one full business cycle. Weekday-only tests miss weekend behavior. Aim for 14 days minimum.
- Segment the results. A test can be flat overall but +25% on mobile. Segment by device, source, and new vs. returning.
- Log every test in a shared doc. Losing tests are just as valuable as winners. Build a testing library your team can search.
- Ship losers too — as data. A "no lift" test still teaches you what does not move the needle. That saves your next test cycle.
Checking a test's p-value every hour and stopping the moment it hits 95% is called peeking. It produces false positives at a rate of 20–30%. Set your sample size in advance and only look at the result once you hit it.
Common A/B testing mistakes to avoid
- Calling winners too early — anything under 1,000 conversions per variant is statistical noise.
- Testing tiny changes only — a 2px padding tweak rarely moves conversion. Test big, bold changes for big learnings.
- Ignoring seasonality — testing a Black Friday hero in January will mislead you.
- Running overlapping tests — two live tests on the same page contaminate each other's results.
- Not logging losers — you will retest the same idea in six months if you do not write it down.
Frequently asked questions
Most tests need 2–4 weeks to reach statistical significance. The exact duration depends on your traffic volume and baseline conversion rate. Ending too early leads to false positives, so wait for 95% confidence and at least one full business cycle.
A/B testing compares two versions with one variable changed. Multivariate testing changes several elements simultaneously and measures every combination. A/B tests need far less traffic and are simpler to interpret. Start with A/B.
Absolutely. Free tools inside email platforms, ad managers, and website builders make testing accessible on any budget. Even testing two subject lines in your next email counts and often reveals a 10–20% lift.
Aim for 1,000–5,000 conversions per variation to hit 95% confidence. Sample size calculators from VWO, Optimizely, or Evan Miller give an exact number based on your baseline rate and the lift you want to detect.
Not when done properly. Serve variants with a 302 redirect or a rel=canonical to the original URL, and never cloak content to Googlebot. Google's guidance is explicit: A/B testing is fine as long as you are not treating search engines differently from users.
Related glossary terms
Sources
