Publications

Educators

2021
Working Paper

Quantifying the Value of Iterative Experimentation

By: Iavor I Bojinov and Jialiang Mao

Over the past decade, most technology companies and a growing number of conventional firms have adopted online experimentation (or A/B testing) into their product development process. Initially, A/B testing was deployed as a static procedure in which an experiment was... View Details

Keywords: Product Development; Value Creation; Research

September 2020 (Revised July 2022)
Exercise

Artea (B): Including Customer-Level Demographic Data

This collection of exercises aims to teach students about 1)Targeting Policies; and 2)Algorithmic bias in marketing—implications, causes, and possible solutions. Part (A) focuses on A/B testing analysis and targeting. Parts (B),(C),(D) Introduce algorithmic bias. The... View Details

Keywords: Targeting; Algorithmic Bias; Race; Gender; Marketing; Diversity; Customer Relationship Management; Demographics; Prejudice and Bias; Retail Industry; Apparel and Accessories Industry; Technology Industry; United States

September 2020 (Revised June 2023)
Supplement

Spreadsheet Supplement to Artea Teaching Note

Spreadsheet Supplement to Artea Teaching Note 521-041. This collection of exercises aims to teach students about 1)Targeting Policies; and 2)Algorithmic bias in marketing—implications, causes, and possible solutions. Part (A) focuses on A/B testing analysis and... View Details

Keywords: Targeted Advertising; Algorithmic Data; Bias; Advertising; Race; Gender; Diversity; Marketing; Customer Relationship Management; Prejudice and Bias; Analytics and Data Science; Retail Industry; Apparel and Accessories Industry; Technology Industry; United States

September 2020 (Revised July 2022)
Exercise

Artea (D): Discrimination through Algorithmic Bias in Targeting

This collection of exercises aims to teach students about 1)Targeting Policies; and 2)Algorithmic bias in marketing—implications, causes, and possible solutions. Part (A) focuses on A/B testing analysis and targeting. Parts (B),(C),(D) Introduce algorithmic bias. The... View Details

Keywords: Targeted Advertising; Discrimination; Algorithmic Data; Bias; Advertising; Race; Gender; Marketing; Diversity; Customer Relationship Management; Prejudice and Bias; Analytics and Data Science; Retail Industry; Apparel and Accessories Industry; Technology Industry; United States

September 2020 (Revised July 2022)
Exercise

Artea (C): Potential Discrimination through Algorithmic Targeting

This collection of exercises aims to teach students about 1)Targeting Policies; and 2)Algorithmic bias in marketing—implications, causes, and possible solutions. Part (A) focuses on A/B testing analysis and targeting. Parts (B),(C),(D) Introduce algorithmic bias. The... View Details

Keywords: Targeting; Algorithmic Bias; Race; Gender; Marketing; Diversity; Customer Relationship Management; Prejudice and Bias; Retail Industry; Apparel and Accessories Industry; Technology Industry; United States

September 2020 (Revised February 2024)
Teaching Note

Artea (A), (B), (C), and (D): Designing Targeting Strategies

Teaching Note for HBS No. 521-021,521-022,521-037,521-043. This collection of exercises aims to teach students about 1)Targeting Policies; and 2)Algorithmic bias in marketing—implications, causes, and possible solutions. Part (A) focuses on A/B testing analysis and... View Details

Keywords: Targeted Advertising; Targeting; Race; Gender; Diversity; Marketing; Customer Relationship Management; Prejudice and Bias; Analytics and Data Science; Retail Industry; Apparel and Accessories Industry; Technology Industry; United States

2024
Working Paper

Anytime-Valid Inference in Linear Models and Regression-Adjusted Causal Inference

By: Michael Lindon, Dae Woong Ham, Martin Tingley and Iavor I. Bojinov

Linear regression adjustment is commonly used to analyze randomized controlled experiments due to its efficiency and robustness against model misspecification. Current testing and interval estimation procedures leverage the asymptotic distribution of such estimators to... View Details

Keywords: Mathematical Methods; Analytics and Data Science

Article

Vungle Inc. Improves Monetization Using Big-Data Analytics

By: Bert De Reyck, Ioannis Fragkos, Yael Grushka-Cockayne, Casey Lichtendahl, Hammond Guerin and Andrew Kritzer

The advent of big data has created opportunities for firms to customize their products and services to unprecedented levels of granularity. Using big data to personalize an offering in real time, however, remains a major challenge. In the mobile advertising industry,... View Details

Keywords: Big Data; Monetization; Data and Data Sets; Advertising; Mobile Technology; Customization and Personalization; Performance Improvement

Find at Harvard

2020
Working Paper

The Effects of Hierarchy on Learning and Performance in Business Experimentation

By: Sourobh Ghosh, Stefan Thomke and Hazjier Pourkhalkhali

Do senior managers help or hurt business experiments? Despite the widespread adoption of business experiments to guide strategic decision-making, we lack a scholarly understanding of what role senior managers play in firm experimentation. Using proprietary data of live... View Details

Keywords: Experimentation; Innovation; Search; New Product Development; Innovation and Invention; Organizational Design; Learning; Performance

2023
Article

Balancing Risk and Reward: An Automated Phased Release Strategy

By: Yufan Li, Jialiang Mao and Iavor Bojinov

Phased releases are a common strategy in the technology industry for gradually releasing new products or updates through a sequence of A/B tests in which the number of treated units gradually grows until full deployment or deprecation. Performing phased releases in a... View Details

Keywords: Product Launch; Mathematical Methods; Product Development