How a holdout proves an ad paid
7 minute read
Attribution says who was there; a test asks what would have happened without the ad. Follow a geo holdout from picking regions to reading the result, and why a small store can get "too small to tell".
The question a test answers
Attribution tells you who was there when a sale happened. An incrementality test asks something harder: how many of those sales would you have lost if the ads had never run?
The gap between the two can be large. Someone searches for your shop by name, clicks your ad, and buys. The ad gets the credit. But they were already on their way. A test counts only the sales that wouldn't have happened without the ad. Those are the incremental ones, and they're what you're really paying for.
Say a campaign is credited with 200 orders a month. A holdout shows that without it you'd still have had 150. The campaign's incremental sales are 50, not 200, and its real return is a quarter of what its dashboard says. Those are sample numbers, but that's the shape of the gap a test can show.
A geo holdout, step by step
A geo holdout uses places as its test groups. You switch an ad campaign off in some regions and leave it on in others, then compare.
- 01Pick regions
- 02Ads off in test
- 03Weekly sales
- 04The lift
- ABon
- IVon
- Goff
- EHon
- NEon
- LAon
- LSoff
- YOon
- Mon
- NGoff
- Bon
- NRon
- CFoff
- BSon
- SOon
- BNoff
Test: 5 regions, ads offControl: 11, ads on
Sales a week, per region
Control regionsTest regionsRange
+£4,400
a month, most likely
between +£2,300 and +£6,900 a month
The whole range is above zero
+£300
a month, most likely
between -£2,100 and +£2,700 a month
Too small to measure.
- Pick regions that behave alike. Match them on past sales, so that before the test the two groups track each other closely.
- Switch the ads off in some. These are the holdout regions. The rest carry on as normal.
- Wait. Leave everything else alone: prices, other campaigns, the website.
- Compare the lines. If the holdout regions fall behind, the gap is what the ads added.
Forge Mark runs geo holdout tests from a wizard and compares orders by the region they shipped to. It writes the result in plain words, and it says so when the store is too small for the test to tell. "Too small to measure" is an answer, not a failure.
How long
Long enough for the gap to show above the normal ups and downs of your sales, and long enough to cover how long your shoppers take to decide. If people usually take a week to buy, a one-week test measures the wrong thing. It would catch the end of last week's decisions and miss this week's.
The sample on this page uses 28 days. Shorter tests are noisier. Longer ones cost more, because every day you hold ads off in the holdout regions is a day you may be giving up sales.
How big
This is the question small stores run into. A test measures a lift with some noise around it, and noise shrinks only as the number of orders grows. The relationship is the square root: 25 times the orders gives a band one fifth as wide.
12 orders a day, 28-day test
Lift from the ads
Too small to tell
+4% measured, somewhere from -13% to +21%
Smallest lift this test could call real: 17%
In the picture, the true lift is the same in all three. The small store (12 orders a day) measures +4%, but its range runs from -13% to +21%, and that range includes zero. The test can't tell a decent lift from none at all. The large store (300 orders a day) measures +9% with a range from +5% to +12%, which is a real answer.
The number at the foot of the picture is the plain-words version of what testers call the smallest detectable lift. It's the smallest measured lift the test could call real. For the small store that's 17%. If the ads really lift sales by 9%, a test this size will usually miss it.
Reading a range
A test gives you a measured lift and a range around it. Read the range first.
- The whole range is above zero. The ads added sales. The measured figure is your best guess for how many, and the ends are the plausible extremes.
- The range crosses zero. You can't say the ads don't work. You can only say this test couldn't tell. Both "no effect" and "a good effect" sit inside the range.
- The range is wide but above zero. The ads worked, but you don't know by how much. Act on the low end.
If the range crosses zero, you have three options. Run the test longer. Test a bigger change, such as switching the whole channel off instead of one campaign. Or lean on the other lenses, such as a marketing mix model, and treat the test as a check.
Calibrating the model
A marketing mix model estimates what each channel added from past weeks. A test measures it directly for one channel in one period. Put them side by side and you learn how far to trust the model.
Say the model said Meta added £4,100 a month and a holdout measured something close, with a range that includes that figure. You can trust both a little more. Say the test measured half that. Trust the test, and ask why the model was off. Often the cause is that Meta's spend moved together with another channel's, so the model couldn't split them.
That's why Forge Mark shows the model and the tests beside each other, in its "Measured by" bar, rather than forcing one answer.
What it can't tell you
- It has a price. In the holdout regions you switch off ads that may be working. If they are, you lose those sales for the length of the test.
- It answers one question. It says what one campaign added, in the regions you chose, in the weeks you ran it. It doesn't say what the campaign adds in December.
- Regions leak. A shopper in a held-out region can see an ad in the next one. That tends to make the measured lift smaller than the true one.
- Other things change. A sale, a stock-out or a viral post in one group but not the other muddies the gap.
None of that is a reason not to test. It's a reason to read a result as a range and to keep the test simple.
Frequently asked questions
How much does a test cost me?
In the regions where ads are off, you give up whatever those ads would have brought in. If the ads work, that's real sales. It's why you hold out a small share of regions for a fixed time.
Can a small store run one?
You can, but the answer may be "too small to tell". Forge Mark says so plainly instead of guessing.
What's a geo test?
A test where the groups are places. Ads are on in some regions and off in others, and you compare the orders each region sends.
Does a test need cookies?
No. Forge Mark compares orders by the region they shipped to, so it doesn't follow any visitor.
Run a test on your own store.
Connect your store in a few minutes, read only, and Forge Mark says whether it has enough orders for a holdout to tell.