Holdout Methodology for Paid Social in Pharma
How to run holdout tests for pharma paid social: platform lift tools, user and geographic holdouts, outcomes, sample size, and privacy.
The short answer
A paid social holdout test randomly withholds ads from part of an audience or some regions and compares outcomes with the exposed group. Pharma brands can use platform lift tools for on-platform and site outcomes, geographic holdouts for broader outcomes, and privacy-safe outcome partners for prescription measures. Plan sample size before launch, keep the holdout clean for the whole test, and make sure health-related audiences and conversion tracking pass privacy review.
Paid social reports can look strong: cheap clicks, high engagement, low cost per site visit. Holdout tests show whether any of that changed outcomes. For pharma, privacy rules shape how they can be run.
Holdout types
| Type | How | Best for |
|---|---|---|
| Platform user-level holdout | Platform randomly withholds ads from some users | Site visits, on-platform actions |
| Geographic holdout | Selected regions get no social ads | Broader outcomes, including Rx via aggregated data |
| Time-based | Turn off in alternating periods | Rarely clean; confounded by seasonality |
Geographic designs avoid user-level tracking and work with aggregated outcome data. Open-source tools such as GeoLift help design and analyze them.
Choosing outcomes
- Site visits or tool use with consented tracking.
- Conversation signals such as discussion guide downloads.
- Prescriptions or starts through privacy-safe, aggregated data for geographic tests.
Avoid sending health information to platforms through conversion pixels without privacy review. See tracking pixel governance.
Designing the test
- Question. "Does paid social add incremental discussion guide downloads and new starts?"
- Unit. Users or regions.
- Holdout size. Use a power calculation based on expected effect and baseline variability.
- Duration. Long enough to capture outcomes, especially prescriptions.
- Clean separation. Make sure the holdout does not get ads through other campaigns on the same platform.
- Analysis plan. Decide the metrics and method before launch.
Reading results
- Report lift with a confidence interval.
- An inconclusive result is a result. It often means the effect is smaller than the test could detect.
- Compare cost per incremental outcome with other channels measured the same way.
Health considerations
- Health audiences on social platforms are restricted by platform policies.
- Lookalike and custom audiences built from health data need privacy review.
- Comments during the test still need adverse event monitoring.
Common mistakes
- Tiny holdouts that cannot detect plausible effects.
- Holdout contaminated by other campaigns.
- Health data in conversion events.
- Declaring failure from an underpowered test.
Practical takeaway
Before your next paid social flight, run a power calculation for a geographic holdout using last year's regional outcome data. If the required holdout is too large to accept, measure a nearer outcome, such as discussion guide downloads, rather than running an underpowered Rx test.
Frequently asked questions
What is a holdout test in paid social?
A test where some eligible people or regions are randomly prevented from seeing ads, so their outcomes can be compared with those who did see them.
Can pharma use platform conversion lift tools?
Sometimes, for site or on-platform outcomes, subject to platform health policies and privacy review of conversion tracking. Prescription outcomes need separate privacy-safe measurement.
How large should a social holdout be?
Large enough to detect the effect you care about. Plan with a power calculation; small holdouts often produce inconclusive results.
Sources
- Meta Open Source, GeoLift (GitHub)
- HHS, Use of Online Tracking Technologies by HIPAA Covered Entities and Business Associates
- American Statistical Association, Statement on p-Values
External guidance and platform documentation change. Links were current at publication; check them again before relying on them for a decision.
Editorial note. Analysis and frameworks are the author's own and do not represent Acxiom or any current or former employer, client, or named platform. Examples labeled hypothetical or illustrative are not results from real campaigns. Nothing here is legal, regulatory, or medical advice.
New pharma programmatic breakdowns, occasionally
One email when I publish something worth reading. Benchmarks, measurement teardowns, and case studies with the caveats attached. No cadence promises, no reselling your address.
Unsubscribe any time. See the privacy policy.
Working through this decision on a real plan?
I work on health and pharma data, identity, and activation, after five years running HCP and DTC programmatic agency-side. Happy to talk through how this applies to your situation.