Publications

2023
Working Paper

An Experimental Design for Anytime-Valid Causal Inference on Multi-Armed Bandits

By: Biyonka Liang and Iavor I. Bojinov

Typically, multi-armed bandit (MAB) experiments are analyzed at the end of the study and thus require the analyst to specify a fixed sample size in advance. However, in many online learning applications, it is advantageous to continuously produce inference on the... View Details

Keywords: Analytics and Data Science; AI and Machine Learning; Mathematical Methods

June 1998
Article

Managing Experimentation in the Design of New Products

By: S. Thomke

Keywords: Design; Product; Management

April 2020
Article

Designs for Estimating the Treatment Effect in Networks with Interference

By: Ravi Jagadeesan, Natesh S. Pillai and Alexander Volfovsky

In this paper, we introduce new, easily implementable designs for drawing causal inference from randomized experiments on networks with interference. Inspired by the idea of matching in observational studies, we introduce the notion of considering a treatment... View Details

Keywords: Experimental Design; Network Inference; Neyman Estimator; Symmetric Interference Model; Homophily

2020
Working Paper

The Effects of Hierarchy on Learning and Performance in Business Experimentation

By: Sourobh Ghosh, Stefan Thomke and Hazjier Pourkhalkhali

Do senior managers help or hurt business experiments? Despite the widespread adoption of business experiments to guide strategic decision-making, we lack a scholarly understanding of what role senior managers play in firm experimentation. Using proprietary data of live... View Details

Keywords: Experimentation; Innovation; Search; New Product Development; Innovation and Invention; Organizational Design; Learning; Performance

04 Dec 2019
Book

Creating the Experimentation Organization

subtle tweaks to everything from varying shades of color to alternative placement of links and menu options for booking properties. It’s part of an innovative culture of experimentation that pervades every aspect of how the company... View Details

Keywords: by Michael Blanding

1995
Dissertation

The Economics of Experimentation in the Design of New Products and Processes

By: Stefan Thomke

Keywords: Economics; Research and Development; Operations; Product Launch

2022
Working Paper

Can a Website Bring Unemployment Down? Experimental Evidence from France

By: Aïcha Ben Dhia, Bruno Crépon, Esther Mbih, Louise Paul-Delvaux, Bertille Picard and Vincent Pons

We evaluate the impact of an online platform giving job seekers tips to improve their search and recommendations of new occupations and locations to target, based on their personal data and labor market data. Our experiment used an encouragement design and was... View Details

Keywords: Online Platform; Digital Platform; Unemployment; Encouragement Design; Job Search; Jobs and Positions; Internet and the Web; Well-being; Outcome or Result; Digital Platforms; France

Register to Read

July 2002
Article

The Economist As Engineer: Game Theory, Experimental Economics and Computation As Tools of Design Economics

By: Alvin E Roth

Keywords: Game Theory; Economics; Mathematical Methods; Design

April 2020
Article

A Theory of Experimenters: Robustness, Randomization, and Balance

By: Abhijit Banerjee, Sylvain Chassang, Sergio Montero and Erik Snowberg

This paper studies the problem of experiment design by an ambiguity-averse decisionmaker who trades off subjective expected performance against robust performance guarantees. This framework accounts for real-world experimenters’ preference for randomization. It also... View Details

Keywords: Experiment Design; Experimenters; Theory; Performance

September 2020 (Revised June 2023)
Exercise

Artea: Designing Targeting Strategies

By: Eva Ascarza and Ayelet Israeli

This collection of exercises aims to teach students about 1)Targeting Policies; and 2)Algorithmic bias in marketing—implications, causes, and possible solutions. Part (A) focuses on A/B testing analysis and targeting. Parts (B),(C),(D) Introduce algorithmic bias. The... View Details

Purchase

2001
Chapter

The Impact of Technology on Knowledge Creation: A Study of Experimentation in Integrated Circuit Design

By: Stefan Thomke

Research Summary

Motivation and Incentive Design

Professor Ashraf's research in motivation and incentives focuses on how to design incentives in sectors where it is important that individuals are motivated by service, such as healthcare or environmental conservation. An important lever is the... View Details

05 Dec 2011
Research & Ideas

It’s Alive! Business Scholars Turn to Experimental Research

you may not be able to learn anywhere else," says Alvin E. Roth, a professor in the Negotiation, Organizations & Markets (NOM) Unit at HBS and a longtime champion of experimental economics. For example, to track how viewers watch... View Details

Keywords: by Carmen Nobel

May 2003
Module Note

Product Development Process Design and Improvement

By: Stefan H. Thomke

Describes the conceptual foundations and pedagogy for a module on the design and improvement of processes for the development of products and services. Can also be used as part of a more general course on Managing Technology and Innovation. Central to Managing Product... View Details

Keywords: Product Development; Design; Performance Improvement; Innovation and Management; Information Technology; Customers; Information Management; Business Processes; Learning; Transformation; Service Operations

October 2018
Case

Booking.com

By: Stefan Thomke and Daniela Beyersdorfer

The case reveals how Booking.com has become the world's leading travel accommodation platform. The company has put online experimentation at the heart of how it designs digital experiences for its customers and partners. To unlock the potential of large-scale testing,... View Details

Keywords: Travel; Product Innovation; Experimentation; A/B Testing; User Experience Design; Product Development; Product Design; Innovation and Management; Transformation; Information Technology; Digital Transformation; Travel Industry

Educators

Purchase

2020
Working Paper

Design and Analysis of Switchback Experiments

By: Iavor I Bojinov, David Simchi-Levi and Jinglong Zhao

In switchback experiments, a firm sequentially exposes an experimental unit to a random treatment, measures its response, and repeats the procedure for several periods to determine which treatment leads to the best outcome. Although practitioners have widely adopted... View Details

Keywords: Switchback Experiments; Design; Analysis; Mathematical Methods