View on GitHub


Online Advertising Incrementality Testing And Experimentation: Industry Practical Lessons

Presented at the KDD 2021



Online advertising has historically been approached as user targeting and ad-to-user matching problems within sophisticated optimization algorithms. As the research area and ad tech industry have progressed over the last couple of decades, advertisers have increasingly emphasized the causal effect estimation of their ads (aka incrementality) using controlled experiments (or A/B testing). Even though observational approaches have been derived in marketing science since the 80s including media mix models, the availability of online advertising personalization has enabled the deployment of more rigorous randomized controlled experiments with millions of individuals.

These evolutions in marketing science, online advertising, and the ad tech industry have posed incredible challenges for engineers, data scientists, and marketers alike. With low effect percentage differences (or lift) and often sparse conversion rates, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in the measurement precision and detailed implementation. Similarly, the correct interpretation of results addressing a business goal within the marketing science domain requires significant data science and experimentation research expertise. All these challenges on the ongoing evolution of the online advertising industry and the heterogeneity of its sources (social, paid search, native, programmatic, etc).

In the current tutorial, we propose a practical, grounded view in the incrementality testing landscape, including:

We will provide first-hand lessons on developing and operationalizing such a platform in a major combined DSP and ad network; these are based on running tens of experiments for up to two months each over the last couple of years.

Outline and Presentations

  1. slides The basics: context and challenges
  2. slides Incrementality Testing: concepts, solutions and literature
  3. slides From concept to production: platform building, challenges, case studies
  4. slides Deployment at Scale: test cycle and case studies
  5. slides Emerging trends: identity challenges, industry trends and solutions

Some of the papers cited in the slides are avaiable at the folder papers

The detailed outline of the tutorial is available here as well as the tutorial ACM paper here.

Feedback welcome!!