Can I Trust a Creative Comparison if the Same Shoppers May Have Seen Both Ads?

Identify the experiment's actual assignment unit

Two advertisements running simultaneously are not necessarily an A/B experiment. They may compete in auctions, reach overlapping audiences and receive different delivery opportunities.

Google Ads documents custom experiments with different traffic-splitting arrangements. For eligible Search experiments, it describes cookie-based and search-based splitting; for Display custom experiments, it documents cookie-based splitting. These mechanisms are not equivalent. A search-based split can assign different searches from the same person to different experiment arms, whereas a cookie-based split is intended to keep a browser's assignments consistent within that experiment.

Inspect the actual experiment type, campaign eligibility, allocation setting and assignment unit. Do not assume that two ordinary ad sets are isolated simply because their audience definitions look similar.

Google's documentation also distinguishes experiments from lift studies, which can use exposed and unexposed groups for a different causal question. A creative-versus-creative comparison should not silently borrow the interpretation of a holdout lift study.

Separate three kinds of exposure overlap

First, investigate within-test crossover: a shopper may see A and B during the same comparison. Second, investigate prior exposure: someone assigned to B may already remember A from an earlier campaign. Third, investigate parallel campaign exposure: another active campaign may show the same product or creative while the experiment is running.

These conditions have different implications. Within-test crossover may violate a planned independent-person comparison. Prior exposure may change familiarity without invalidating every descriptive observation. Parallel campaigns may make it difficult to attribute a response to one tested asset.

Record what the advertising platform actually exposes. If person-level exposure histories are unavailable, mark them unknown; do not reconstruct individual identities from private data or claim the platform guarantees isolation that its documentation does not establish.

Review cookies, browsers and devices as possible assignment limitations. A cookie-based split should not be described as verified cross-device person-level isolation without corresponding evidence.

Hypothetical lamp creative comparison

Hypothetical illustration—not an executed advertising experiment: a fictional lamp merchant compares creative A, a clean product photograph, with creative B, the same lamp in a desk setting. Both have the same verified offer.

The merchant initially creates two independently running campaigns with overlapping audience definitions. The worksheet classifies this as an observational delivery comparison, not a confirmed randomized experiment.

A proposed revision uses an eligible platform experiment with a documented split setting. The reviewer records which assignment unit the chosen experiment actually supports and checks whether an older lamp campaign remains active.

If the older campaign is still distributing A, prior or parallel exposure remains possible even when the new experiment is configured correctly. The merchant can document that limitation, revise the campaign arrangement where appropriate, or narrow the conclusion to the observed conditions.

No impressions, orders, statistical results or actual audience overlaps are asserted.

Decide what the comparison can legitimately establish

Use the experiment's actual reporting and uncertainty information. Keep the primary outcome, eligible population, exposure window and attribution definitions consistent with the assignment method.

If overlap cannot be ruled out, do not describe the results as an uncontaminated comparison of two independent shopper groups. You may still report observed campaign outcomes, but state the exposure limitation and avoid assigning the difference solely to the artwork.

Do not remove existing customers from every campaign merely to manufacture a supposedly clean experiment. The right population depends on the business question and the available experiment design.

CartVisual can supply candidate visual assets; audience assignment, exposure records and measurement occur in the advertising platform. The practical next step is to verify the chosen experiment's splitting contract before spending effort interpreting a winner.

Record your own observations

The filled rows are illustrative examples. Use the blank row for your own files and observations.

Download the editable CSV worksheet

What the references establish

• Google Ads Help — Set Up a Custom Experiment: Documents experiment traffic allocation, cookie-based versus search-based splitting, Display cookie splitting and interpretation limits when campaigns change.
• Google Ads Help — About the Google Ads Experiment Center: Distinguishes randomized campaign experiments from controlled lift-study methodologies.