Should I Randomize Shoppers, Products or Ad Sets When Testing a New Product-Photo Style Across a Catalog?

Write the decision before choosing a unit

Begin with one precise question. Are you asking whether a new image-production method preserves product accuracy across difficult SKUs? Or whether a new approved photo style changes a shopper outcome?

The first is a production pilot. The existing fixed-product-test-set guide addresses it by holding source references and tasks constant and recording factual defects. Its unit may be a product, an image-generation attempt or an approved asset.

The second is a campaign experiment. Its unit must correspond to the intended exposure comparison and the outcome being measured. Google's experiment documentation describes supported traffic-splitting methods, but not every campaign type offers the same assignment controls.

Do not call a catalog pilot a shopper-level A/B test simply because half the SKUs receive one visual style and half receive another. Different products may have different prices, audiences, demand and purchase cycles.

Understand what each assignment choice changes

Shopper or browser assignment: a suitable supported experiment can compare different creative treatments within an eligible population. Check whether the platform actually assigns people, cookies, searches or another unit. The distinction affects possible crossover.

Product or SKU assignment: one group of products receives style A and another receives B. This may be useful for operational rollout, but a difference in sales can reflect product identity rather than visual style. A popular product and an obscure product are not interchangeable experimental units.

Ad-set or campaign assignment: campaigns may differ in audience, bidding, placement or optimization. Simply creating two ad sets does not establish random assignment or equivalent opportunities.

For a catalog-wide treatment, also consider interference. A shopper may see several products from the same family, or a campaign may promote multiple related SKUs. Assigning neighboring products to different treatments can make the intended comparison harder to interpret.

Record the unit, grouping rule and known overlap routes. Do not assume that product-family grouping automatically removes all interference.

Hypothetical six-SKU lamp catalog

Hypothetical illustration—not an executed production or advertising experiment: a fictional merchant has six lamps across two product families.

The first proposed plan assigns three SKUs to clean-background images and three to lifestyle images, then compares aggregate orders. The plan is rejected as a causal creative test because the products themselves differ and no shopper assignment has been established.

A second plan is a production-quality pilot: each SKU receives an approved clean-background reference and a proposed lifestyle-style candidate. Reviewers record whether the candidate preserves the lamp's shape, finish, controls and included components. This can answer an asset-quality question without measuring buyer response.

A third plan, still unexecuted, asks whether the advertising platform can support a controlled creative comparison for the same eligible products and offers. Its actual assignment unit, audience and reporting contract must be verified before launch.

The worksheet distinguishes these plans rather than combining them into one invented experiment.

Record interference and analyze at the intended level

Keep the assigned unit attached to every observation. If entire product families are assigned together, do not analyze every impression as though it were an independently randomized family. If shoppers are assigned to variants, do not claim SKU-level independence merely because the report contains many product rows.

For consequential causal conclusions, use a suitable statistical design and qualified review. Clustered assignments and repeated shopper exposures can require methods different from a simple independent-impression comparison.

If the platform cannot implement the intended design, narrow the claim. A production pilot can still identify label errors, poor crops and inconsistent lighting. An observational campaign comparison can describe its results without attributing them solely to the image style.

Do not replace one product's approved photo with an inaccurate image for experimental purity. Product fidelity is a prerequisite, not an outcome to trade against clicks.

The practical next step is to select either a controlled asset-quality pilot or a supported shopper-response experiment, and maintain separate records for the two.

Record your own observations

The filled rows are illustrative examples. Use the blank row for your own files and observations.

Download the editable CSV worksheet

What the references establish

• Google Ads Help — Set Up a Custom Experiment: Documents supported campaign experiments, traffic-split methods and limitations of changing campaign conditions.
• Google Ads Help — About the Google Ads Experiment Center: Distinguishes randomized experiment designs and controlled lift-study methodologies. It does not establish SKU-level randomization as a universal platform feature.