What establishes incremental marketing impact?

Counterfactuals, advertising experiments and the assumptions behind observational measurement

This retrospective review considers source versions available by 24 October 2025. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: causal inference; incrementality; experiments; advertising; counterfactuals

1. What does it mean for marketing to cause an outcome?

A customer can see an advertisement and buy the product without the advertisement causing the purchase. The causal question concerns what would have happened to that customer, or to a comparable population, under a different advertising condition. This counterfactual is the centre of incrementality measurement. More detailed attribution data can describe a sequence of contacts while leaving the counterfactual unresolved.

AI changes the quantity and speed of marketing decisions, but it does not remove this identification problem. A model can generate more variants, select audiences or predict conversion probabilities. The effect of those decisions still depends on a comparison that separates their contribution from the reasons people were likely to buy anyway. Accurate prediction of a buyer and accurate estimation of an intervention's effect are different statistical tasks.

This review asks what established causal-inference and advertising-measurement research permits analysts to conclude from experiments, geographic comparisons and observational models. It focuses on the inference connecting a marketing intervention to outcomes. The productivity of producing the intervention and the quality of its creative execution are separate questions. The distinction matters because a cheaper campaign can be less effective, and a more effective campaign can still be unprofitable once all relevant costs are included.

2. Review method and inferential scope

Targeted searches on 21 October 2026 located twelve primary methodological papers and advertising field studies through publisher, author-institution and research-organisation records. Selection covered counterfactual reasoning, experimental precision, observational bias, geographic experiments, synthetic controls, time-series methods, staggered adoption and external validity. The empirical descriptions are limited to inspected abstracts, research summaries and bibliographic records. Full statistical models, supplementary materials and underlying datasets were not independently reviewed.

The synthesis is conceptual and comparative. No effect sizes are pooled across campaigns or translated into a general return on advertising. A study about search advertising does not automatically estimate the effect of video, display or a brand campaign. A methodological paper establishes conditions for an estimator; it does not show that those conditions hold in a particular advertiser's data. Maintaining that distinction allows the methods to be compared without assigning an artificial universal ranking.

The discussion of AI-supported marketing is an application of established identification questions, rather than a new empirical claim about AI. Where a platform chooses who receives an intervention, where a team changes several practices together or where the outcome occurs after a delay, the treatment and counterfactual must be defined accordingly. These situations can require different designs even when a dashboard gives all of them the same label of campaign performance.

3. Potential outcomes and the treatment definition

Rubin's foundational account formalises causal effects using the contrast between treatment outcomes and the outcomes that would occur under an alternative condition [1]. Because both cannot be observed for the same unit in the same circumstances, design and assumptions determine how the missing counterfactual is estimated. Randomisation is valuable because assignment does not deliberately select units according to their potential outcomes.

The treatment must be specific enough to interpret. Switching on AI-generated copy while also changing audience targeting and media spend estimates the effect of a package. It cannot isolate the effect of the copy by itself. A comparison can still answer a useful package-level question, but the conclusion should name the package. Retrospectively assigning all observed change to the most visible new component introduces a claim the design never tested.

The unit also matters. A user-level intervention concerns assigned users, a store-level intervention concerns stores and a regional intervention concerns geographic units. Thousands of transactions within one treated region do not create thousands of independently randomised regions. The unit used to analyse uncertainty should reflect how treatment was assigned and how outcomes may be related. Otherwise a large transaction count can give an unjustified appearance of precision.

An intention-to-treat comparison uses the assigned condition, including units that did not actually receive or use the intervention. Restricting attention to people who engaged with an advertisement changes the population after assignment. That engaged group may differ in prior interest. A causal claim about exposure itself requires additional assumptions or design features beyond observing engagement among assigned users.

4. What large advertising experiments reveal

Lewis and Rao examine 25 large advertising field experiments and show that even experiments reaching very large audiences can leave returns imprecisely measured [2]. Advertising effects may be small relative to variation in purchasing. A large dataset is therefore insufficient evidence of a precise commercial estimate. The relevant quantity is the signal relative to the noise and the design's ability to distinguish them.

Gordon and colleagues compare observational estimates with randomised results from 15 Facebook advertising experiments. Their institutional summary reports that observational methods often failed to recover the experimental effects despite extensive demographic and behavioural information [3]. This is direct evidence against assuming that granular covariates automatically remove exposure selection. It does not establish that every observational estimate is wrong or that the same discrepancy applies to every future platform.

Blake, Nosko and Tadelis use a large paid-search field experiment to study consumer heterogeneity [4]. The study's question makes explicit that response can differ across customers. A campaign-level average can combine people whose purchases are responsive to advertising with people who would reach the seller anyway. Generalising a result requires attention to the mix of those populations, rather than treating an average as a fixed property of a channel.

These studies also distinguish statistical and managerial questions. An estimate may be positive while its uncertainty includes commercially unattractive values. Another estimate may be too imprecise to exclude a useful effect. A decision can still be required, but uncertainty should not be converted into a categorical finding of success or failure simply because a budget decision is binary.

5. Randomised designs and operational constraints

Kohavi and colleagues describe the design of online controlled experiments, including randomisation, sample size, variance reduction and organisational limitations [11]. Their account treats experimentation as an implemented system with technical failure modes. A label of A/B test does not guarantee that assignment, exposure logging and outcome collection functioned as intended.

For marketing, contamination can arise when people in the control condition encounter the treatment through another route. Cross-device use, shared households or broad media coverage can alter the meaning of a clean individual comparison. The design must state whether the estimand concerns assignment to a channel, opportunity to see a message or receipt of a specific exposure. These are related interventions with different practical meanings.

Ghost-ad methodology addresses difficulties in constructing informative advertising controls by recording opportunities associated with the ad delivery process [5]. The paper is relevant because the ad auction and targeting system form part of the selection process. Comparing served users with arbitrary unserved users can confound the effect of advertising with the selection that determines service. The method's applicability depends on the platform implementation; its existence does not imply that every advertiser has the necessary data.

Repeated experimentation introduces additional measurement choices. Looking at many outcomes, stopping when a favourable result appears or selecting a winning subgroup after inspecting the data can change the interpretation of uncertainty. A report should distinguish a planned primary comparison from exploratory results. The latter may justify further testing, but they do not inherit the evidential status of a prospectively specified test merely because they were found in the same dataset.

6. Geographic experiments and aggregate outcomes

Vaver and Koehler describe geo experiments in which non-overlapping geographic regions are randomly assigned to advertising treatment or control [6]. This creates a comparison at the geographic level and can be useful where individual-level intervention is unavailable. Its validity concerns the way regional assignments translate into different advertising conditions, as well as how outcomes are recorded.

Geography changes the practical sources of variation. Regions can differ in size, baseline demand, competitive activity and seasonal patterns. More transactions within a region improve measurement of that region's outcome but do not eliminate differences between regions. A design with few regions may have limited information about between-region variability even when its total revenue is large.

The outcome window must also match the response being studied. A short window can capture immediate transactions while missing delayed effects. A long window may admit additional interventions and environmental changes. There is no universally correct duration independent of the causal question. The report should identify whether it estimates an immediate response, a cumulative effect over a specified period or persistence after treatment ends.

A geographic result may include multiple mechanisms. Advertising can influence existing customers, attract new customers or shift the timing and location of purchases. A sales difference estimates the combined effect on that measured outcome. Explaining which mechanism produced it requires further observations. A causal estimate of total impact should not be presented as a proven account of every behavioural step within it.

7. Modelled counterfactuals and their assumptions

Brodersen and colleagues propose Bayesian structural time-series models to predict a counterfactual market response, allowing local trends, seasonality and covariates. They illustrate the method with simulated data and an advertising application [7]. The model provides a structured way to estimate what an untreated trajectory might have been. The causal interpretation still depends on whether the predictors and intervention setting justify that counterfactual.

Synthetic control methods construct a comparison from weighted untreated units. Abadie, Diamond and Hainmueller develop this approach for comparative case studies and apply it to California's tobacco-control programme [8]. Its relevance to marketing is methodological. A good pre-intervention fit is informative, but a comparison can cease to represent the untreated outcome if the intervention affects the donor units or if their relationship changes for other reasons.

Predictive accuracy in the pre-period and causal validity after the intervention are related but distinct. A forecasting model can fit the past using a variable that itself changes because of advertising. Conditioning on that variable after treatment can remove part of the effect being estimated. Selecting controls therefore requires substantive reasoning about their relationship to the intervention, rather than only a search for the strongest historical correlation.

Uncertainty intervals produced by a model are conditional on its structure and assumptions. They do not automatically quantify uncertainty about an omitted competitor action, a misrecorded launch date or contamination of the control. Sensitivity analysis examines how conclusions depend on such choices. It is especially useful when several plausible specifications answer slightly different questions, but it cannot create an untreated comparison where none is credible.

The Brodersen manuscript explicitly assumes that control covariates remain unaffected by treatment. Its advertising application compares estimates using randomised control regions with estimates using observational industry-keyword searches. It also tests for an apparent effect in untargeted regions [7]. These checks investigate the construction of the counterfactual within that application. Their value is diagnostic: agreement there supports that case, while leaving the suitability of another campaign's covariates to be established.

8. Staggered adoption, trends and correlated outcomes

Callaway and Sant'Anna address difference-in-differences with multiple periods, different treatment dates and conditional parallel-trend assumptions. Their framework separates group-time effects and alternative aggregation schemes [9]. This is relevant when marketing systems or AI workflows roll out in stages. A single before-after coefficient can mix cohorts with different exposure histories and different effects.

Bertrand, Duflo and Mullainathan examine the reliability of inference in difference-in-differences settings [10]. The paper motivates attention to the dependence of outcomes across time. Repeated monthly observations are not necessarily independent pieces of information. Ignoring their relationship can make uncertainty appear smaller than the data justify. The reviewed record supports this methodological concern, without supplying a universal correction for every panel structure.

A credible comparison requires reasoning about untreated trends. Two groups can begin at different outcome levels and still have comparable untreated changes; groups with similar levels can face different trends. Pre-period observations can investigate the plausibility of the assumption, but a failure to detect a pre-trend does not prove that the assumption holds. The amount of available pre-period information and the precision of the test also matter.

Rollout timing can itself be selected. Teams may adopt a tool first where performance is weak, leadership is enthusiastic or the required data are already available. Those reasons can also affect later outcomes. Calling the rollout natural does not remove selection. A defensible study specifies what variation identifies the effect and why the remaining differences do not explain it.

9. External validity and AI-supported decisions

Allcott's work on site-selection bias shows why the settings that adopt or evaluate a programme can differ from the settings to which its results are later applied [12]. In marketing, a willing early adopter may have cleaner data, more experienced staff or a different customer base. A successful trial establishes evidence about the tested setting. Transporting it requires examining the differences that could alter the response.

AI-assisted optimisation adds a related interpretation problem. A system can improve a prediction and still choose an intervention with a weak causal effect. High-propensity customers may be easy to predict but difficult to influence. Evaluation should therefore distinguish a prediction target from the outcome contrast used to estimate incremental response. The distinction follows directly from the counterfactual question; it is not a claim that every optimisation system ignores incrementality.

A campaign can also improve one measured outcome while changing another. More clicks can coexist with unchanged sales, and a short-term conversion gain can leave longer-term brand outcomes unmeasured. The evaluation needs an explicit outcome hierarchy and observation period. Adding a business label to a proxy does not supply the missing evidence connecting it to profit or future demand.

Economic interpretation adds another boundary. Incremental revenue is not incremental profit unless the costs of supplying the additional purchases are accounted for. A return calculated on media expenditure alone answers a different question from a return calculated on the full campaign investment. If AI reduces production costs while the audience effect is unchanged, the economic return can improve without a larger treatment effect on purchasing. Conversely, a larger sales response can be offset by higher fulfilment or acquisition costs. Reporting the causal effect and the economic calculation separately keeps those mechanisms observable.

10. Synthesis and limitations

Established causal research provides several routes to incremental marketing measurement, each with a defined comparison and assumptions. Randomisation can address assignment selection, while observational and aggregate methods can be informative where their identifying conditions are defensible. Large datasets, strong predictions and detailed attribution do not independently establish causality.

This review uses summary-level access and selective coverage. It does not reanalyse campaigns, reproduce uncertainty estimates or assess the suitability of a particular platform's experimental implementation. Historical field studies identify persistent measurement problems, rather than a current universal advertising return. Their channel, population and treatment boundaries remain material to interpretation.

For AI-supported marketing, the central implication is that the production method and the causal evaluation are separate parts of the evidence. An intervention should be defined, its counterfactual justified and its outcome measured over a relevant period. The resulting estimate can then support a bounded claim about incrementality. Its precision and external validity determine how far that claim can travel.

Source access and appraisal register

The 12 references below were located through targeted title and topic searches on 21 October 2026. Author or publisher abstracts, official research summaries and publication records were consulted. The discussion uses the claims available in those materials; full manuscripts, supplementary analyses and underlying data were not independently appraised. Institutional guidance is treated as guidance, rather than a controlled estimate of effectiveness. Where a working paper is cited, its provisional status remains part of the interpretation. The evidence map is selective and does not estimate the prevalence of results in the complete literature.

The Brodersen et al. full manuscript was additionally inspected for the control-covariate assumption and the advertising application comparing randomised and observational controls (Sections 2 and 4). The remaining sources were assessed at abstract or record level.

References