Iavor I. Bojinov

Iavor I. Bojinov

Assistant Professor of Business Administration
Richard Hodgson Fellow

Assistant Professor of Business Administration
Richard Hodgson Fellow

Technology and Operations Management
+1 (617) 496-2024
 
Iavor I. Bojinov
Unit
Contact Information
(617) 496-2024
Featured Work Publications Research Summary
Overview
Over the last decade, technology companies like Amazon, Google, and Netflix have pioneered data-driven research and development processes centered on massive experimentation. However, as companies increase the breadth and scale of their experiments to millions of interconnected customers, existing statistical methods have become inadequate, causing inefficiencies and biased results. The bias is often substantial enough to change the magnitude and sign of the results, leading managers to make incorrect decisions — such as releasing inferior products and dropping promising initiatives. The inefficiencies are similarly costly, slowing the innovation process and inadvertently overexposing customers to harmful changes or needlessly delaying beneficial offerings.

My research develops novel statistical methodologies to address these challenges and enable managers to experiment more rigorously, safely, and efficiently in modern business contexts.

Rigor: Traditional statistical theory overlooks three fundamental factors that, when ignored, lead to biased results. First, incorporating time is essential as customers arrive sequentially, not in batches, and past changes can have a prolonged impact. Second, customers often interact directly (through communications) or indirectly by, for example, competing for a limited resource (like riders vying for drivers on Lyft), so changes to one person’s experience can interfere with the outcomes of others. My research focuses on developing methods for designing and analyzing experiments that incorporate time and accommodate interference by either grouping units to limit the interference or by adjusting for it in the analysis, ensuring that the results are unbiased and robust.
Examples of academic papers:
  • Han, K. W., Basse, G., and Bojinov, I. (2024). Population Interference in Panel Experiments. Journal of Econometrics, 238(1), 105565.
  • Bojinov, I., and Shepherd, N. (2019). Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading. Journal of the American Statistical Association, 14(528), 1665-1682.

Safety: Managers use experimentation to de-risk the innovation process by limiting their customers’ exposure to negative changes; for example, the Google Play and Apple App stores launch all changes to applications using a sequence of experiments known as a phased release. My work provides frameworks for managers to balance the inherent trade-off between the risk of releasing a negative change and the desire to learn causal effects.
Examples of academic papers:
  • Li, Y., Mao, J., and Bojinov, I. (2023). Balancing Risk and Reward: An Automated Phased Release Strategy. Advances in Neural Information Processing Systems, 36.
  • Woong Ham, D., Lindon, M., Tingley, M., and Bojinov, I. Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation.

Efficiency: As companies integrate experimentation into their innovation process, managers increasingly seek ways to improve the efficiency of their experiments by achieving the same precision with fewer participants. This is especially important for multi-sided platform companies, like Doordash and Uber, that use complex experimental designs to overcome interference. My work provides novel experimental designs that draw on optimization techniques to achieve the same precision with far fewer participants, drastically increasing efficiency and reducing the cost of the experiment.
Examples of academic papers:
  • Bojinov, I., Simchi-Levi, D., and Zhao, J. (2023). Design and Analysis of Switchback Experiments. Management Science 69 (7), 3759-3777.
  • Ni, T., Bojinov, I., and Zhao, J. Design of Panel Experiments with Spatial and Temporal Interference. Available at SSRN 4466598.

Currently, I am particularly interested in studying the application of experimentation to the operationalization of artificial intelligence (AI), the process by which AI products are developed and integrated into real-world applications. Two idiosyncrasies make experimentation particularly challenging in this context. First, AI products often interact with other algorithms, products, or systems, causing unintended consequences. Second, AI products are built by iterating between experimentation and development, allowing managers to identify improvement opportunities that often lead to product changes during the experiment.
Examples of academic papers:
  • Rajkumar K., Saint-Jacques G., Bojinov I., Brynjolfsson E., and Aral S. (2022). A Causal Test of the Strength of Weak Ties. Science 377.6612: 1304-1310.
  • Yue, D., Hamilton, P., and Bojinov, I. Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development.

Much of my work in this area has been summarized in the following practitioner-focused articles:
  • Bojinov, I. (2023) Keep Your AI Projects on Track. Harvard Business Review 101, (6): 5359.
  • Bojinov I. and Gupta S. (2022) Online Experimentation: Benefits, Operational and Methodological Challenges, and Scaling Guide. Harvard Data Science Review, 4(3).
  • Bojinov, I., Saint-Jacques, G., and Tingley, M., (2020) Avoid the Pitfalls of A/B Testing, Harvard Business Review 98 (2), 48-53.

Iavor Bojinov is an Assistant Professor of Business Administration and the Richard Hodgson Fellow at Harvard Business School. He is the co-PI of the AI and Data Science Operations Lab and a faculty affiliate in the Department of Statistics at Harvard University and the Harvard Data Science Initiative. His research focuses on developing novel statistical methodologies to make business experimentation more rigorous, safer, and efficient, specifically homing in on the application of experimentation to the operationalization of artificial intelligence (AI), the process by which AI products are developed and integrated into real-world applications. His work has been published in top academic journals such as Annals of Applied Statistics, Biometrika, The Journal of the American Statistical Association, The Journal of Econometrics, Quantitative Economics, Management Science, and Science, and has been cited in Forbes, The New York Times, The Washington Post, and Reuters, among other outlets. More broadly, as one of the few scholars who work at the intersection of data science and business, he was the first author to have spotlight featured articles in both the Harvard Business Review and the Harvard Data Science Review.

Professor Bojinov is also the co-creator of the first-year required MBA course “Data Science for Managers” and has previously taught the “Competing in the Age of AI” and “Technology and Operations Management” courses. Before joining Harvard Business School, Professor Bojinov worked as a data scientist leading the causal inference effort within the Applied Research Group at LinkedIn. He holds a Ph.D. and an MA in Statistics from Harvard and an MSci in Mathematics from King’s College London.

Featured Work
Population Interference in Panel Experiments
The phenomenon of population interference, where a treatment assigned to one experimental unit affects another experimental unit’s outcome, has received considerable attention in standard randomized experiments. The complications produced by population interference in this setting are now readily recognized, and partial remedies are well known. Less understood is the impact of population interference in panel experiments where treatment is sequentially randomized in the population, and the outcomes are observed at each time step. This paper proposes a general framework for studying population interference in panel experiments and presents new finite population estimation and inference results. Our findings suggest that, under mild assumptions, the addition of a temporal dimension to an experiment alleviates some of the challenges of population interference for certain estimands. In contrast, we show that the presence of carryover effects — that is, when past treatments may affect future outcomes — exacerbates the problem. Our results are illustrated through both an empirical analysis and an extensive simulation study.
Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading
We define causal estimands for experiments on single time series, extending the potential outcome framework to dealing with temporal data. Our approach allows the estimation of a broad class of these estimands and exact randomization-based p-values for testing causal effects, without imposing stringent assumptions. We further derive a general central limit theorem that can be used to conduct conservative tests and build confidence intervals for causal effects. Finally, we provide three methods for generalizing our approach to multiple units that are receiving the same class of treatment, over time. We test our methodology on simulated “potential autoregressions,” which have a causal interpretation. Our methodology is partially inspired by data from a large number of experiments carried out by a financial company who compared the impact of two different ways of trading equity futures contracts. We use our methodology to make causal statements about their trading methods. Supplementary materials for this article are available online.
Balancing Risk and Reward: An Automated Phased Release Strategy
Phased releases are a common strategy in the technology industry for gradually releasing new products or updates through a sequence of A/B tests in which the number of treated units gradually grows until full deployment or deprecation. Performing phased releases in a principled way requires selecting the proportion of units assigned to the new release in a way that balances the risk of an adverse effect with the need to iterate and learn from the experiment rapidly. In this paper, we formalize this problem and propose an algorithm that automatically determines the release percentage at each stage in the schedule, balancing the need to control risk while maximizing ramp-up speed. Our framework models the challenge as a constrained batched bandit problem that ensures that our pre-specified experimental budget is not depleted with high probability. Our proposed algorithm leverages an adaptive Bayesian approach in which the maximal number of units assigned to the treatment is determined by the posterior distribution, ensuring that the probability of depleting the remaining budget is low. Notably, our approach analytically solves the ramp sizes by inverting probability bounds, eliminating the need for challenging rare-event Monte Carlo simulation. It only requires computing means and variances of outcome subsets, making it highly efficient and parallelizable.
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation.
Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers' decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to "peek" at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-arm bandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.
Design and Analysis of Switchback Experiments
Switchback experiments, where a firm sequentially exposes an experimental unit to random treatments, are among the most prevalent designs used in the technology sector, with applications ranging from ride-hailing platforms to online marketplaces. Although practitioners have widely adopted this technique, the derivation of the optimal design has been elusive, hindering practitioners from drawing valid causal conclusions with enough statistical power. We address this limitation by deriving the optimal design of switchback experiments under a range of different assumptions on the order of the carryover effect—the length of time a treatment persists in impacting the outcome. We cast the optimal experimental design problem as a minimax discrete optimization problem, identify the worst-case adversarial strategy, establish structural results, and solve the reduced problem via a continuous relaxation. For switchback experiments conducted under the optimal design, we provide two approaches for performing inference. The first provides exact randomization-based p-values, and the second uses a new finite population central limit theorem to conduct conservative hypothesis tests and build confidence intervals. We further provide theoretical results when the order of the carryover effect is misspecified and provide a data-driven procedure to identify the order of the carryover effect. We conduct extensive simulations to study the numerical performance and empirical properties of our results and conclude with practical suggestions.
Design of Panel Experiments with Spatial and Temporal Interference
One of the main practical challenges companies face when running experiments (or A/B tests) over a panel is interference, the setting where one experimental unit's treatment assignment at one time period impacts another's outcomes, possibly at the following time period. Existing literature has identified aggregating units into clusters as the gold standard to handle interference, yet the degree of aggregation remains an open question. In this work, we present a new randomized design of panel experiments and answer this question when all experimental units are modeled as vertices on a two-dimensional grid. Our proposed design has two features: the first feature is a notion of randomized spatial clustering that randomly partitions units into equal-size clusters; the second is a notion of balanced temporal randomization that extends the classical completely randomized designs to the temporal interference setting. We prove the theoretical performance of our design, develop its inferential techniques, and verify its superior performance by conducting an extensive simulation study.
A Causal Test of the Strength of Weak Ties
The strength of weak ties is an influential social-scientific theory that stresses the importance of weak associations (e.g., acquaintance versus close friendship) in influencing the transmission of information through social networks. However, causal tests of this paradoxical theory have proved difficult. Rajkumar et al. address the question using multiple large-scale, randomized experiments conducted on LinkedIn’s “People You May Know” algorithm, which recommends connections to users (see the Perspective by Wang and Uzzi). The experiments showed that weak ties increase job transmissions, but only to a point, after which there are diminishing marginal returns to tie weakness. The authors show that the weakest ties had the greatest impact on job mobility, whereas the strongest ties had the least. Together, these results help to resolve the apparent “paradox of weak ties” and provide evidence of the strength of weak ties theory.
Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development
Predictive model development is understudied despite its importance to modern businesses. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms) as the primary driver of model quality, the value of tools that implement those methods has been neglected. In a field experiment leveraging a predictive data science contest, we study the importance of tools by restricting access to software libraries for machine learning models. By only allowing access to these libraries in our control group, we find that teams with unrestricted access perform 30% better in log-loss error — a statistically and economically significant amount, equivalent to a 10-fold increase in the training data set size. We further find that teams with high general data-science skills are less affected by the intervention, while teams with high tool-specific skills significantly benefit from access to modeling libraries. Our findings are consistent with a mechanism we call ‘Tools-as-Skill,’ where tooling automates and abstracts some general data science skills but, in doing so, creates the need for new tool-specific skills.
Keep Your AI Projects on Track
AI—and especially its newest star, generative AI—is today a central theme in corporate boardrooms, leadership discussions, and casual exchanges among employees eager to supercharge their productivity. Sadly, beneath the aspirational headlines and tantalizing potential lies a sobering reality: Most AI projects fail. Some estimates place the failure rate as high as 80%—almost double the rate of corporate IT project failures from a decade ago. Approaches exist, however, to increase the odds of success. Companies can greatly reduce their risk of failure by carefully navigating five critical steps that every AI project traverses on its way to becoming a product: selection, development, evaluation, adoption, and management.
Importance of Being Causal

Causal inference is the study of how actions, interventions, or treatments affect outcomes of interest. The methods that have received the lion’s share of attention in the data science literature for establishing causation are variations of randomized experiments. Unfortunately, randomized experiments are not always feasible for a variety of reasons, such as an inability to fully control the treatment assignment, high cost, and potential negative impacts. In such settings, statisticians and econometricians have developed methods for extracting causal estimates from observational (i.e., nonexperimental) data. Data scientists’ adoption of observational study methods for causal inference, however, has been rather slow and concentrated on a few specific applications. In this article, we attempt to catalyze interest in this area by providing case studies of how data scientists used observational studies to deliver valuable insights at LinkedIn. These case studies employ a variety of methods, and we highlight some themes and practical considerations. Drawing on our learnings, we then explain how firms can develop an organizational culture that embraces causal inference by investing in three key components: education, automation, and certification.


Online Experimentation: Benefits, Operational and Methodological Challenges, and Scaling Guide
In the past decade, online controlled experimentation, or A/B testing, at scale has proved to be a significant driver of business innovation. The practice was first pioneered by the technology sector and, more recently, has been adopted by traditional companies undergoing a digital transformation. This article provides a primer to business leaders, data scientists, and academic researchers on business experimentation at scale, explaining the benefits, challenges (both operational and methodological), and best practices in creating and scaling an experimentation-driven, decision-making culture.
Avoid the Pitfalls of A/B Testing

The use of online A/B testing has spread rapidly in recent years, fueled by the growing appreciation of its value and the relatively low costs and increasing availability of technology needed to conduct them. Today, it is no exaggeration to say that the successful application of A/B testing is critical to their futures. But many firms often make inadvertent mistakes in how they conduct these experiments. In this article—which employs examples from Netflix and LinkedIn—we offer strategies that companies can adopt to avoid them so they can more effectively spot new opportunities and threats and improve the long-term performance of their businesses.

Journal Articles
Working Papers
Cases and Teaching Materials
Research Summary
Overview
Over the last decade, technology companies like Amazon, Google, and Netflix have pioneered data-driven research and development processes centered on massive experimentation. However, as companies increase the breadth and scale of their experiments to millions of interconnected customers, existing statistical methods have become inadequate, causing inefficiencies and biased results. The bias is often substantial enough to change the magnitude and sign of the results, leading managers to make incorrect decisions — such as releasing inferior products and dropping promising initiatives. The inefficiencies are similarly costly, slowing the innovation process and inadvertently overexposing customers to harmful changes or needlessly delaying beneficial offerings.

My research develops novel statistical methodologies to address these challenges and enable managers to experiment more rigorously, safely, and efficiently in modern business contexts.

Rigor: Traditional statistical theory overlooks three fundamental factors that, when ignored, lead to biased results. First, incorporating time is essential as customers arrive sequentially, not in batches, and past changes can have a prolonged impact. Second, customers often interact directly (through communications) or indirectly by, for example, competing for a limited resource (like riders vying for drivers on Lyft), so changes to one person’s experience can interfere with the outcomes of others. My research focuses on developing methods for designing and analyzing experiments that incorporate time and accommodate interference by either grouping units to limit the interference or by adjusting for it in the analysis, ensuring that the results are unbiased and robust.
Examples of academic papers:
  • Han, K. W., Basse, G., and Bojinov, I. (2024). Population Interference in Panel Experiments. Journal of Econometrics, 238(1), 105565.
  • Bojinov, I., and Shepherd, N. (2019). Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading. Journal of the American Statistical Association, 14(528), 1665-1682.

Safety: Managers use experimentation to de-risk the innovation process by limiting their customers’ exposure to negative changes; for example, the Google Play and Apple App stores launch all changes to applications using a sequence of experiments known as a phased release. My work provides frameworks for managers to balance the inherent trade-off between the risk of releasing a negative change and the desire to learn causal effects.
Examples of academic papers:
  • Li, Y., Mao, J., and Bojinov, I. (2023). Balancing Risk and Reward: An Automated Phased Release Strategy. Advances in Neural Information Processing Systems, 36.
  • Woong Ham, D., Lindon, M., Tingley, M., and Bojinov, I. Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation.

Efficiency: As companies integrate experimentation into their innovation process, managers increasingly seek ways to improve the efficiency of their experiments by achieving the same precision with fewer participants. This is especially important for multi-sided platform companies, like Doordash and Uber, that use complex experimental designs to overcome interference. My work provides novel experimental designs that draw on optimization techniques to achieve the same precision with far fewer participants, drastically increasing efficiency and reducing the cost of the experiment.
Examples of academic papers:
  • Bojinov, I., Simchi-Levi, D., and Zhao, J. (2023). Design and Analysis of Switchback Experiments. Management Science 69 (7), 3759-3777.
  • Ni, T., Bojinov, I., and Zhao, J. Design of Panel Experiments with Spatial and Temporal Interference. Available at SSRN 4466598.

Currently, I am particularly interested in studying the application of experimentation to the operationalization of artificial intelligence (AI), the process by which AI products are developed and integrated into real-world applications. Two idiosyncrasies make experimentation particularly challenging in this context. First, AI products often interact with other algorithms, products, or systems, causing unintended consequences. Second, AI products are built by iterating between experimentation and development, allowing managers to identify improvement opportunities that often lead to product changes during the experiment.
Examples of academic papers:
  • Rajkumar K., Saint-Jacques G., Bojinov I., Brynjolfsson E., and Aral S. (2022). A Causal Test of the Strength of Weak Ties. Science 377.6612: 1304-1310.
  • Yue, D., Hamilton, P., and Bojinov, I. Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development.

Much of my work in this area has been summarized in the following practitioner-focused articles:
  • Bojinov, I. (2023) Keep Your AI Projects on Track. Harvard Business Review 101, (6): 5359.
  • Bojinov I. and Gupta S. (2022) Online Experimentation: Benefits, Operational and Methodological Challenges, and Scaling Guide. Harvard Data Science Review, 4(3).
  • Bojinov, I., Saint-Jacques, G., and Tingley, M., (2020) Avoid the Pitfalls of A/B Testing, Harvard Business Review 98 (2), 48-53.
Additional Information
Areas of Interest
In The News

In The News

More News for Iavor I. Bojinov
Additional Information

Areas of Interest

In The News

More News for Iavor I. Bojinov