Online Advertising Incrementality Testing: Practical Lessons, Paid Search and Emerging Challenges

Presented at the ECIR 2022 in person!!

Authors

• Joel Barajas, Amazon, Marketing Measurement(work done while employed at Yahoo Research)
• Narayan Bhamidipati, Yahoo Research, yahoo!
• James G. Shanahan, Church and Duncan Group Inc., UC Berkeley

Abstract

Online advertising has historically been approached as an ad-to-user matching problem within sophisticated optimization algorithms. As the research and ad-tech industries have progressed, advertisers have increasingly emphasized the causal effect estimation of their ads (incrementality) using controlled experiments (A/B testing).

With low lift effects and sparse conversion, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in measurement precision. Similarly, the correct interpretation of results addressing a business goal requires significant data science and experimentation research expertise.

We propose a practical tutorial in the incrementality testing landscape, including:

• The business need
• Literature solutions and industry practices
• Designs in the development of testing platforms
• The testing cycle, case studies, and recommendations
• Emerging trends: identity challenges, paid search, industry trends and solutions

We provide first-hand lessons based on the development of such a platform in a major combined DSP and ad network, and after running several tests for up to two months each over recent years.

We elaborate more on the user privacy implications in online experimentation and incrementality testing. We aim to motivate the research community to focus on solutions under these emerging constraints.

Outline and Presentations

1. slides The basics: context and challenges
2. slides Incrementality Testing: concepts, solutions and literature
3. slides From concept to production: platform building, challenges, case studies
4. slides Deployment at Scale: test cycle and case studies
5. slides Emerging trends: identity challenges, paid search, industry trends and solutions

Some of the papers cited in the slides are avaiable at the folder papers

The tutorial conference submission, which includes the detailed outline of the tutorial, is available at the conference proceedings here. We keep a local copy in this page repository here.

Feedback welcome!!