Publications

March–April 2020
Article

Avoid the Pitfalls of A/B Testing

By: Iavor I. Bojinov, Guillaume Sait-Jacques and Martin Tingley

Online experiments measuring whether “A,” usually the current approach, is inferior to “B,” a proposed improvement, have become integral to the product-development cycle, especially at digital enterprises. But often firms make serious mistakes in conducting these... View Details

Keywords: A/B Testing; Experiment Design; Social Networks; Product Development; Performance Improvement; Measurement and Metrics; Social Media

Register to Read

October 2020
Background Note

The When and How of Effective Beta Testing

By: Robert J. Dolan

Keywords: Beta; Testing

2020
Working Paper

Digital Experimentation and Startup Performance: Evidence from A/B Testing

By: Rembrand Koning, Sharique Hasan and Aaron Chatterji

Recent work argues that experimentation is the appropriate framework for entrepreneurial strategy. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of experimentally testing... View Details

Keywords: Experimentation; A/B Testing; Data-driven Decision-making; Entrepreneurship; Strategy; Business Startups; Information Technology; Performance

SSRN

Read Now

10 Dec 2014
News

Yes, A/B Testing Is Still Necessary

Keywords: statistics; experimentation; testing; products

September 2022
Article

Experimentation and Start-up Performance: Evidence from A/B Testing

By: Rembrand Koning, Sharique Hasan and Aaron Chatterji

Recent scholarship has argued that experimentation should be the organizing principle for entrepreneurial strategy. Experimentation leads to organizational learning, which drives improvements in firm performance. We investigate this proposition by exploiting the... View Details

Keywords: Experimentation; A/B Testing; Data-driven Decision-making; Organizational Learning; Entrepreneurship; Strategy; Business Startups; Learning; Performance; Decision Making

Read Now

October 2018 (Revised January 2019)
Case

The Financial Crisis: Timothy Geithner and the Stress Tests

By: Samuel G. Hanson, Robin Greenwood, David Scharfstein and Adi Sunderam

In February and March 2009, the U.S. economy was in the midst of a terrifying financial and economic crisis. Between the beginning of 2008 and early 2009, four of the 25 largest U.S. financial institutions had failed, and nine of these 25 institutions had taken... View Details

Keywords: Bailout; Regulation; Stress Test; Financial Crisis; History; Economy; Policy; Governing Rules, Regulations, and Reforms; Decision Making; Banking Industry; Financial Services Industry; Real Estate Industry; United States

November 1989 (Revised March 1992)
Background Note

Concept Testing

By: Robert J. Dolan

Describes concept testing products. Presents guidelines for effective design, execution, and interpretation of test procedures. Discusses limitations of these techniques and sets out the situations for which they are appropriate. View Details

Keywords: Marketing; Mathematical Methods; Forecasting and Prediction

16 Sep 2024
Video

Standardized Tests

Article

Testing Substitutability

By: John William Hatfield, Nicole Immorlica and Scott Duke Kominers

We provide an algorithm for testing the substitutability of a length-N preference relation over a set of contracts X in time O(|X|3⋅N3). Access to the preference relation is essential for this result: We show that a substitutability-testing algorithm with access only... View Details

Keywords: Substitutability; Matching; Communication Complexity; Preference Elicitation; Marketplace Matching; Communication; Mathematical Methods; Economics

Video

September–October 2017
Article

The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests

By: Ron Kohavi and Stefan Thomke

In the fast-moving digital world, even experts have a hard time assessing new ideas. Case in point: At Bing, a small headline change an employee proposed was deemed a low priority and shelved for months until one engineer decided to do a quick online controlled... View Details

Keywords: Experiments; A/B Testing; Research; Consumer Behavior

Register to Read

04 Sep 2015
HBS Conference

Test Conference

20 Aug 2015
News

Companies Like Amazon Need to Run More Tests on Workplace Practices

Article

The Impact of Penalties for Wrong Answers on the Gender Gap in Test Scores

By: Katherine B. Coffman and David Klinowski

Multiple-choice exams play a critical role in university admissions across the world. A key question is whether imposing penalties for wrong answers on these exams deters guessing from women more than men, disadvantaging female test-takers. We consider data from a... View Details

Keywords: Behavioral Economics; Standardized Testing; Gender; Higher Education; Prejudice and Bias

Item

Standardized Tests

As you prepare for your PhD application, watch this video to learn more about what standardized tests are required for admission. Topics covered include the Graduate Record Examination (GRE), Graduate Management Admissions View Details

September 2011
Article

On Testing Business Models

By: D. Huelsbeck, K. Merchant and Tatiana Sandino

This study explored management decisions regarding formal empirical testing of business models. It documented a test of one company's business model under seemingly favorable conditions for such a test – a successful single product firm following a consistent strategy... View Details

Keywords: Performance Measurement; Non-financial Performance Measures; Business Models; Management Control; Decisions; Business Model; Performance Evaluation

SSRN

October 2018 (Revised September 2020)
Case

Testing Autonomy in Pittsburgh

By: Mitchell Weiss and Brittany Urick

Pittsburgh’s mayor had been among the first to welcome self-driving vehicles but was now one of many needing to react after a pedestrian fatality involving an autonomous Uber in Arizona. He had originally preferred to roll out “the red carpet” instead of the “red... View Details

13 Jan 2021
News

Stress Test

lock, but the company soon expanded its focus to B2B sales in the health care sector. Its clients now include sites responsible for COVID-19 testing and vaccine distribution, which require both security and a reliable audit log. The... View Details

September 1994 (Revised May 2010)
Case

Hamilton Test Systems, Inc.

By: William A. Sahlman and Norman Klein

The protagonists must decide whether to invest in an auto emissions testing company as the first investment in the leveraged buyout fund they recently formed. Issues of how to raise the needed equity capital and how to structure the acquisition are emphasized. View Details

Keywords: Venture Capital; Leveraged Buyouts; Environmental Sustainability; Investment; Service Industry; Auto Industry