Publications

2019
Article

Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading

By: Iavor I Bojinov and Neil Shephard

We define causal estimands for experiments on single time series, extending the potential outcome framework to dealing with temporal data. Our approach allows the estimation of a broad class of these estimands and exact randomization based p-values for testing causal... View Details

Keywords: Causality; Nonparametric; Potential Outcomes; Trading Costs; Mathematical Methods

Find at Harvard

Register to Read

2023
Article

Balancing Risk and Reward: An Automated Phased Release Strategy

By: Yufan Li, Jialiang Mao and Iavor Bojinov

Phased releases are a common strategy in the technology industry for gradually releasing new products or updates through a sequence of A/B tests in which the number of treated units gradually grows until full deployment or deprecation. Performing phased releases in a... View Details

Keywords: Product Launch; Mathematical Methods; Product Development

2024
Working Paper

Design of Panel Experiments with Spatial and Temporal Interference

By: Tu Ni, Iavor Bojinov and Jinglong Zhao

One of the main practical challenges companies face when running experiments (or A/B tests) over a panel is interference, the setting where one experimental unit's treatment assignment at one time period impacts another's outcomes, possibly at the following time... View Details

Keywords: Research

26 Apr 2023
In Practice

Is AI Coming for Your Job?

prudent approach for knowledge workers is to harness the potential of AI as a complementary tool, amplifying their capabilities and adapting to the evolving landscape of work. Iavor View Details

Keywords: by Kristen Senz; Technology

Research Summary

Overview

By: Iavor I. Bojinov

Over the last decade, technology companies like Amazon, Google, and Netflix have pioneered data-driven research and development processes centered on massive experimentation. However, as companies increase the breadth and scale of their experiments to millions of... View Details

March 2022
Article

Estimating the Effectiveness of Permanent Price Reductions for Competing Products Using Multivariate Bayesian Structural Time Series Models

By: Fiammetta Menchetti and Iavor Bojinov

Researchers regularly use synthetic control methods for estimating causal effects when a sub-set of units receive a single persistent treatment, and the rest are unaffected by the change. In many applications, however, units not assigned to treatment are nevertheless... View Details

Keywords: Causal Inference; Partial Interference; Synthetic Controls; Bayesian Structural Time Series; Mathematical Methods

Find at Harvard

November–December 2023
Article

Keep Your AI Projects on Track

By: Iavor Bojinov

AI—and especially its newest star, generative AI—is today a central theme in corporate boardrooms, leadership discussions, and casual exchanges among employees eager to supercharge their productivity. Sadly, beneath the aspirational headlines and tantalizing potential... View Details

Keywords: Generative Models; AI and Machine Learning; Success; Failure; Product Development; Technology Adoption

Find at Harvard

Register to Read

August 2021 (Revised February 2024)
Case

Data Science at the Warriors

By: Iavor I. Bojinov and Michael Parzen

The case explores the development and early growth of a data science team at the Golden State Warriors, an NBA team based in San Francisco. The case begins by explaining the initial rationale for investing in data science, then covers a debate on the appropriate team... View Details

Keywords: Data Science; Digital Marketing; Analysis; Forecasting and Prediction; Technological Innovation; Information Technology; Sports Industry; San Francisco; United States

Educators

Purchase

2023
Working Paper

Virtual Water Coolers: A Field Experiment on the Role of Virtual Interactions on Organizational Newcomer Performance

By: Prithwiraj Choudhury, Jacqueline N. Lane and Iavor Bojinov

Designing management practices to better onboard organizational newcomers working remotely is a key priority for firms. We report results from a randomized field experiment conducted at a large global firm that estimates the performance effects of different types of... View Details

Keywords: Remote Work; Virtual Water Coolers; Social Interactions; Careers; Field Experiment; Employees; Interpersonal Communication; Internet and the Web; Performance; Personal Development and Career

2023
Working Paper

Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development

By: Daniel Yue, Paul Hamilton and Iavor Bojinov

Predictive model development is understudied despite its centrality in modern artificial intelligence and machine learning business applications. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms)... View Details

Keywords: Analytics and Data Science

2023
Working Paper

Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation

By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor Bojinov

Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to... View Details

Keywords: Performance Evaluation; Research and Development; Analytics and Data Science; Consumer Behavior

2021
Working Paper

Quantifying the Value of Iterative Experimentation

By: Iavor I Bojinov and Jialiang Mao

Over the past decade, most technology companies and a growing number of conventional firms have adopted online experimentation (or A/B testing) into their product development process. Initially, A/B testing was deployed as a static procedure in which an experiment was... View Details

Keywords: Product Development; Value Creation; Research

2024
Working Paper

Winner Take All: Exploiting Asymmetry in Factorial Designs

By: Matthew DosSantos DiSorbo, Iavor I. Bojinov and Fiammetta Menchetti

Researchers and practitioners have embraced factorial experiments to simultaneously test multiple treatments, each with different levels. With the rise of technologies like Generative AI, factorial experimentation has become even more accessible: it is easier than ever... View Details

Keywords: Factorial Designs; Fisher Randomizations; Rank Estimators; Employer Interventions; Causal Inference; Mathematical Methods; Performance Improvement

January 2024
Article

Population Interference in Panel Experiments

By: Kevin Wu Han, Guillaume Basse and Iavor Bojinov

The phenomenon of population interference, where a treatment assigned to one experimental unit affects another experimental unit’s outcome, has received considerable attention in standard randomized experiments. The complications produced by population interference in... View Details

Keywords: Outcome or Result; Research; Situation or Environment