Measurement · also called Control group
Holdout group
A holdout group is a randomly chosen portion of an audience deliberately not shown a campaign's ads, used as a baseline to measure the ads' true incremental effect.
To know what an ad changed, you need to know what would have happened without it. A holdout group provides that answer: a slice of the target audience, picked at random, that is kept from seeing the ads while everyone else can.
Comparing the holdout with the exposed group reveals incrementality. If both convert at similar rates, the ads did little; if the exposed group converts much more, the ads worked. Holdouts are the backbone of lift studies, and can be user-level (specific IDs held out), geographic (whole regions), or time-based.
The group must be truly random and large enough for statistical confidence, and it must genuinely not see the ads, which is hard across multiple platforms. Some advertisers use a "public service announcement" or ghost-ad approach so the holdout group is shown an unrelated ad, keeping auction dynamics similar. Holdouts cost some short-term sales, but many teams consider that price worth paying for trustworthy measurement.
Think of it like this
A holdout group is like the untreated plants in a garden experiment: without them, you cannot tell whether the fertiliser or the sunshine made the tomatoes grow.
An example
A subscription streaming service in Canada withholds its email and paid social ads from 5% of lapsed subscribers; after a month, 6.1% of the exposed group resubscribe versus 4.9% of the holdout.
Related terms
Incrementality
Incrementality is the extra result, such as sales, installs or visits, caused by advertising that would not have happened without it, usually measured by comparing exposed and unexposed groups.
Lift study
A lift study is an experiment measuring how much a campaign changed an outcome, such as awareness or sales, by comparing exposed people with a similar unexposed group.
Marketing mix modeling (MMM)
Marketing mix modeling (MMM) is a statistical method that estimates how much each marketing channel, plus factors like price and seasonality, contributed to sales, using aggregate historical data.
Attribution
Attribution is the process of assigning credit for a conversion, such as a sale, sign-up or install, to the ads, channels or touchpoints that a customer encountered beforehand.
Campaign
A campaign is an organised set of ads with a shared goal, budget, audience and time period, run across one or more channels and measured against defined KPIs.
Sources: IAB guidelines