Growth & Analytics — July 20267 min read · Part of series: MER vs ROAS: Which Metric Should D2C Brands Actually Use?

How to Run an Incrementality Test for Your D2C Brand: The Step-by-Step Guide

Your Google Brand Search campaign shows 5.2x ROAS. It's your strongest campaign by far. Your agency loves it. You've been increasing budget on it every month.

High Incrementality

> 70%

Genuinely new demand

Recommended Window

14–21 Days

Full purchase cycle

Brand Search Gap

30% – 60%

Typical real incrementality

Here's the uncomfortable question: what would happen if you turned it off?

If the answer is "revenue would drop by exactly what Google is attributing" — the campaign is genuinely incremental. Every conversion it claims is a conversion that wouldn't have happened without it.

If the answer is "revenue would barely move because those customers would have found you on Google organically anyway" — you've been paying for conversions that were going to happen regardless. Your 5.2x ROAS is real. The incrementality behind it is not.

An incrementality test is the only way to find out which one is true. And for most D2C brands, the answer is somewhere in between — and worth knowing precisely.

Attribution tells you who took credit. Incrementality tells you who deserves it.

What Is an Incrementality Test?

An incrementality test (also called a holdout test or lift test) measures the true causal impact of an advertising channel or campaign by comparing what happened with the advertising against what would have happened without it.

Instead of asking "how much revenue did this campaign generate?" (which platforms answer with attribution) it asks "how much revenue would we have lost without this campaign?" (which only a controlled test can answer).

The difference between those two questions is the incrementality gap, and it's where attribution inflation lives.

Why Most D2C Brands Don't Run Incrementality Tests (And Why They Should)

The objection is always the same: "I can't pause a campaign that's working. What if revenue drops?"

This fear is real. But consider the alternative: spending ₹2L/month on a Google Brand Search campaign that has 60% incrementality means ₹80,000/month is being paid to capture conversions that would have happened through organic search anyway. Over a year, that's ₹9.6L in recoverable budget.

The test costs you a two-week pause on one campaign. The insight is worth months of budget optimisation.

The brands that never test never know. And not knowing means making allocation decisions based entirely on which platform is best at claiming credit, not which platform is actually driving growth.

The 3 Types of Incrementality Tests

Type 1: Channel Holdout Test

Pause an entire channel for a defined period and measure the impact on total Shopify revenue.

Best for: Testing whether a channel is driving genuinely incremental revenue or largely capturing organic demand (common test for Google Brand Search, email, and SMS).

How it works: Pause the channel completely. Monitor total Shopify revenue for 14 days. Compare against the equivalent 14-day period before the test (ideally the same weekdays to control for day-of-week effects).

What you're measuring: Revenue drop as a percentage of what the channel was attributing.

Type 2: Geographic Holdout Test

Continue running your full campaign in most regions but pause it in a specific geographic test region. Compare revenue trends in the holdout region against regions where campaigns ran normally.

Best for: Testing channels where a full pause would be too risky or disruptive to operations. Allows you to test incrementality without stopping all activity.

How it works: Select 2–3 cities or states that represent approximately 10–15% of your total revenue. Pause the channel in those regions. Run for 14–21 days. Compare revenue change in holdout regions against change in control regions.

Type 3: Audience Holdout Test

Within a specific campaign, randomly suppress the ads for a subset of your target audience (the holdout group) while continuing to show ads to the rest (the test group). Compare conversion rates between the two groups.

Best for: Testing whether a retargeting or remarketing campaign is actually driving incremental conversions or just showing ads to people who were going to buy anyway.

How it works: Meta's native Holdout Experiments tool (in Ads Manager → Experiments → Holdout) allows you to set a holdout percentage (typically 10–20% of the target audience sees no ads). After 7–14 days, compare the conversion rate of the holdout group against the ad-exposed group. The lift = true incrementality.

Channel Holdout Test Framework and Calculations

Step-by-Step: Running a Channel Holdout Test

This is the most practical incrementality test for most D2C brands — straightforward to set up and provides clear, actionable data.

Step 1: Choose What to Test First

Start with the channel that shows the highest ROAS and therefore has the most to lose if incrementality is poor. The highest-ROAS channels are the ones most likely to be taking credit for organic demand.

Most common first tests for D2C brands:

Google Brand Search — typically 30–60% true incrementality. Organic search captures much of what paid brand search claims.
Email/SMS remarketing — typically 40–70% true incrementality. Customers who were going to repurchase anyway trigger email conversions.
Meta retargeting — typically 50–80% true incrementality, varies significantly by window length and audience size.

💡 Start with Google Brand Search. It's the most commonly over-attributed channel and has the clearest test structure (pause brand keyword bidding, monitor organic brand search traffic and conversions).

Step 2: Establish Your Baseline

Before pausing, record for the 14 days immediately before your test:

Metric	14-Day Baseline
Total Shopify net revenue	₹X
Shopify net revenue by day	Daily breakdown (to spot day-of-week patterns)
Total new customers acquired	N
Spend on the channel being tested	₹Y
Channel-attributed revenue (from platform)	₹Z

⚠️ Note: Run the test during a stable, non-promotional period. Avoid the 2 weeks before or after a major sale, festival, or product launch — external factors make it impossible to isolate the channel's impact.

Step 3: Pause the Channel

For Google Brand Search: pause all campaigns with brand keywords. Leave all non-brand campaigns running unchanged.
Set a clear test window: 14 days minimum. 21 days is better for channels where purchase cycles extend beyond a week.
Communicate internally: tell your team the test is running. The instinct to "fix" dropping metrics mid-test will destroy the data.

Step 4: Measure During the Test

Monitor daily:

Metric	What to Watch
Shopify net revenue (daily)	Primary outcome metric
Organic search traffic	Should increase as paid brand search stops — organic captures some
Direct traffic	Often increases as branded searches shift to direct navigation
Other channels' attributed revenue	Should not change significantly — large changes suggest contamination

Do not make any other significant changes during the test period. No budget changes to other campaigns, no new creative launches, no promotions. Isolating the variable is everything.

Step 5: Calculate Incrementality

After the test period:

Revenue drop = Baseline daily revenue − Test period daily revenue

Channel-attributed revenue per day (from platform) = Pre-test attribution ÷ 14 days

Incrementality % = Revenue Drop ÷ Channel-Attributed Daily Revenue × 100

Numerical Example:

Google Brand Search pre-test: ₹2,40,000/month attributed (₹8,000/day)
Baseline Shopify net revenue: ₹52,000/day average
Test period Shopify net revenue: ₹48,500/day average
Revenue drop: ₹3,500/day
Incrementality = ₹3,500 ÷ ₹8,000 = 43.75%

Google Brand Search is 43.75% incremental. This means:

43.75% of what it was attributing represents genuinely incremental revenue
56.25% was capturing conversions that would have happened through organic anyway

At ₹8,000/day spend, you're paying for 56% non-incremental conversions. The budget reclamation opportunity is significant.

Step 6: Apply the Finding

If incrementality is above 70%: The channel is genuinely driving strong new demand. Continue spending. If anything, consider increasing budget.
If incrementality is 40–70%: Mixed signal. Some real demand creation, some organic capture. Consider reducing brand search spend and investing the reclaimed budget in non-brand search (which tends to have higher incrementality) or prospecting.
If incrementality is below 40%: The channel is primarily capturing demand that would have converted anyway. Significantly reduce spend. The organic equivalent (organic search, direct navigation) is doing the heavy lifting — you're paying for a small additive effect on top.

Meta Native Holdout Experiment and Testing Cadence

Running Meta's Native Holdout Experiment

For testing Meta retargeting or prospecting campaigns without a full channel pause:

Go to Meta Ads Manager → Experiments → Create Experiment
Select "Holdout Test"
Choose the campaign(s) you want to test
Set holdout size: 15–20% (this % of your target audience will see no ads from these campaigns)
Set duration: 14–21 days
Launch

Meta automatically tracks the conversion rate difference between the holdout group (no ads) and the exposed group (saw your ads). At the end of the test, Meta reports the lift — the incremental conversion rate increase from showing the ads.

A lift of 30% means your retargeting campaign is driving 30% more conversions than the audience would have generated without seeing the ads. The remaining 70% of attributed conversions were going to happen anyway.

Building an Incrementality Testing Cadence

Incrementality isn't a one-time test. Channels change. Audiences evolve. A channel that was 75% incremental 18 months ago may be 45% incremental today as organic brand awareness has grown.

Recommended cadence:

Quarterly

Test your highest-spend, highest-ROAS channel. The one most worth validating.

Biannually

Run holdout experiments on your Meta retargeting campaigns. Retargeting incrementality shifts significantly with changes in organic traffic volume and brand awareness.

Annually

Full incrementality audit across all channels. Reallocate budget based on what's genuinely driving growth vs capturing organic demand.

Conclusion

Incrementality testing is the most rigorous way to answer the question MER can suggest but not fully resolve: which channels are actually creating revenue, not just claiming it?

The test is simple. The pause is uncomfortable. The data is irreplaceable.

Run your first holdout test this quarter. Start with Google Brand Search. Measure for 14 days. Apply the incrementality percentage to your attribution and discover what your budget is actually earning versus what the platform is claiming.

Attribution tells you who took credit. Incrementality tells you who deserves it.

Key Takeaways

Start with High ROAS Channels: Google Brand Search is often only 30-60% incremental.
14 to 21 Day Test Window: Necessary to cover a full customer consideration cycle.
Reallocate Savings to Prospecting: Move captured-organic spend to high-lift demand creation.

Frequently Asked Questions

What is an incrementality test in marketing?

An incrementality test measures the true causal impact of an ad campaign by comparing what happened with advertising against what would have happened without it. It answers whether a campaign is creating new demand or capturing demand that would have converted anyway the distinction that attribution cannot make.

How long should an incrementality test run?

Minimum 14 days for most D2C purchase cycles. 21 days is better for products with longer consideration cycles (above ₹1,500 AOV) or retargeting campaigns with 30-day windows. The test must run long enough to capture a complete purchase cycle for your typical customer.

What is a good incrementality percentage?

Above 70% is strong most of what the channel is attributing is genuinely incremental. 40–70% is mixed real demand creation exists but significant organic capture is happening. Below 40% is concerning the channel is primarily capturing conversions that would have occurred through other channels anyway.

Won't I lose revenue during the test?

Possibly, but only the incremental portion. If Google Brand Search is 40% incremental, pausing it causes a 40% revenue drop on that channel's attribution. But that means 60% of what it was attributing was already being captured elsewhere. The test cost is the temporary loss of 40% incremental revenue from that channel for 14 days typically worth far more in budget efficiency insights.

How does Flable AI use incrementality data?

Flable's MER calculation already uses Shopify net revenue as the ground truth, providing the aggregate picture. Incrementality tests at the channel level tell you how to allocate within that aggregate. Together, MER + incrementality data + Flable's per-campaign CM2 give the most complete picture of where each marketing rupee is genuinely earning its place.