Incrementality testing is a controlled experiment that measures the true causal impact of a marketing campaign by comparing a test group that sees the campaign against a control group (holdout) that does not. It answers one question: would these conversions have happened without the ad spend? Most attribution models cannot answer this — incrementality testing can.
A 2023 Nielsen study found 60% of marketers could not confidently measure whether their paid media drove real results or just captured people who were already going to convert. Incrementality testing is the methodology that closes that gap.
What is incrementality testing?
Incrementality testing is a randomised controlled experiment for marketing. You split your audience into two groups at random: the test group receives the campaign as normal, while the holdout group (also called the control group) is withheld from the campaign entirely. After the test period, you compare conversion rates between the two groups.
The difference in conversion rates — adjusted for group size — is your incremental lift. If the test group converts at 4% and the holdout at 3.2%, the campaign produced 0.8 percentage points of genuine lift, or roughly 25% incremental improvement.
This is fundamentally different from attribution, which observes the same people and assigns credit backward. Incrementality testing uses a control group, so causality is measurable, not modelled.
Retargeting campaigns routinely show 6–10x attributed ROAS because they target users who were already close to converting. Incrementality tests consistently show the true lift is 1.5–2x. The difference is "credit capture" — the campaign reached people who would have converted regardless.
Why incrementality testing matters for marketing ROI
Without incrementality data, budget allocation is based on attribution credit rather than causal proof. Four reasons this matters at scale:
- Reveals authentic ROAS. Attribution models inflate retargeting and branded search because they capture near-certain converters. Incrementality tests strip away that noise and show what the campaign actually caused.
- Eliminates wasted spend. Brands consistently discover that 10–30% of their paid budget generates zero incremental lift. Every dollar saved there can fund channels that do drive real results.
- Builds executive confidence. A controlled experiment is a more defensible proof of value than a dashboard showing attributed revenue. Leadership teams respond to causality, not correlation.
- Improves quarterly budget allocation. When you know which channels produce genuine lift, you can scale them with confidence and cut channels that only claim credit.
How incrementality testing works
Every incrementality test follows three stages regardless of channel or budget size.
- Design the experiment. Choose the channel or tactic to test (e.g. Facebook retargeting, Google Display, email re-engagement). Define your success metric — usually conversions or revenue. Randomly split your audience into test and control groups. Random assignment is non-negotiable: self-selection bias will corrupt the result.
- Run the holdout period. The test group receives the campaign normally. The holdout group sees no ads from this campaign (they may see a PSA or nothing, depending on the platform). Run the test for at least 2–4 weeks, or one full conversion cycle for longer-consideration products.
- Measure the lift. Compare conversion rates:
(test conversion rate - holdout conversion rate) / holdout conversion rate. This is your incremental lift percentage. Multiply by the holdout group's baseline conversions to calculate the number of conversions the campaign actually caused.
Test group conversion rate: 4.0%
Holdout group conversion rate: 3.2%
→ Incremental lift = (4.0 - 3.2) / 3.2 = 25%
# 25% of test-group conversions were caused by the campaign
Types of incrementality tests — which to use
| Test type | How it works | Best for | Minimum scale |
|---|---|---|---|
| Audience holdout | Random % of audience excluded from campaign | Paid social, email, display | 10,000+ users per group |
| Geo holdout | Campaign runs in some regions, excluded from others | Local businesses, OOH, TV, radio | Matched city pairs |
| Time-based holdout | Campaign paused for defined period, then restarted | Search, email | High daily conversion volume |
| Platform ghost bidding | Platform withholds ads from holdout at auction level | Meta, Google built-in lift tests | Platform minimums apply |
Real incrementality testing examples
Incrementality tests routinely produce results that contradict attributed performance — and that is the point.
1. E-commerce Facebook retargeting
A direct-to-consumer brand spending $25,000 per month on Facebook retargeting saw attributed ROAS of 6x in their ads manager. An audience holdout test over three weeks revealed the true incremental ROAS was 1.8x — the vast majority of converters would have purchased regardless of the retargeting ad. The brand reduced retargeting spend by 40% with no measurable drop in revenue.
2. Local service business geo test
A plumbing company ran paid search in Denver but not in Colorado Springs (matched control market). Denver leads came in 22% higher than Colorado Springs during the test period, and the markets had been within 3% of each other for the prior six months. The test confirmed paid search was driving genuine new business, not just capturing branded searches.
3. B2B content marketing holdout
A SaaS company tracked which leads read 3 or more blog articles before signing up. A holdout group received only email nurtures with no content. Leads who consumed 3+ articles converted to paid at 2.4x the holdout rate, confirming content investment was causal, not merely correlated with buyer intent.
Incrementality testing vs. attribution — which tells the truth?
Both methods have a role, but they answer different questions.
Incrementality testing
- Proves causality — controlled experiment
- Reveals true ROAS free from attribution bias
- Requires a holdout group (some revenue risk)
- Takes 2–4 weeks minimum per test
- Best for budget allocation decisions
Attribution modelling
- Shows correlation across all touchpoints
- Available in real-time from ad platforms
- Cannot separate causality from coincidence
- Consistently overstates retargeting value
- Best for day-to-day optimisation within a channel
6 best practices for running incrementality tests
- Randomise at the right level. For audience tests, randomise by user ID, not cookie — cookies reset too often. For geo tests, match control markets by population, seasonality, and historical conversion rate.
- Size your groups correctly. You need enough conversions in each group to detect a meaningful difference. Run a statistical power calculation before launch: aim for 80%+ power at 95% confidence.
- Do not touch the test mid-flight. Adjusting bids, targeting, or budget during a holdout test corrupts the result. Lock the setup and wait.
- Run one variable at a time. Testing Facebook retargeting and email simultaneously makes it impossible to isolate the individual contribution of each.
- Account for halo effects. A strong TV or social campaign can lift conversion rates in holdout groups through brand awareness. Build holdout periods that precede any major campaign launches.
- Test on a rolling schedule. Consumer behaviour changes quarterly. A test that showed 30% lift in Q4 may show 10% in Q1. Retest channels at least annually.
A brand that tests Facebook retargeting in isolation may see 25% lift, then cuts Google display and sees Facebook lift drop to 8%. The channels were working together — removing one broke the sequence. Incrementality tests work best when they account for cross-channel effects, either through Marketing Mix Modelling or multi-channel holdout designs.
Common incrementality testing mistakes to avoid
- Too-short test periods — two weeks is the minimum; short-cycle products may need more for statistical significance.
- Non-random holdout selection — if your holdout skews toward users who were less likely to convert anyway, the lift estimate is inflated.
- Ignoring the cost of the holdout — withholding ads from potential buyers has a real revenue cost. Factor this into your decision to test.
- Confusing statistical significance with business significance — a 2% incremental lift may be statistically significant but too small to justify the campaign cost.
- Testing only retargeting — prospecting campaigns need incrementality tests too; they are often under-credited by attribution models.
Frequently asked questions
A/B testing compares two variations to find which performs better. Incrementality testing measures "something vs. nothing" — whether a campaign produces real lift above the baseline conversion rate that would have happened anyway.
Most tests need 2–4 weeks minimum to reach statistical significance. The exact duration depends on your conversion volume and average sales cycle length.
Yes. Geo-based holdout tests work well for local businesses with multiple service areas. You need enough conversion volume in each region to achieve statistical validity.
Any positive lift above zero means the campaign drove real results. Brands typically expect 10–40% incremental lift from well-targeted paid channels. If lift is near zero, the spend may not be justified.
Attribution models credit the last touch or distribute credit across touchpoints, but they cannot isolate causality. Retargeting and branded search capture people who would have converted anyway, so attributed ROAS is almost always higher than true incremental ROAS.
Related glossary terms
Sources
