Publications

March–April 2020
Article

Avoid the Pitfalls of A/B Testing

By: Iavor I. Bojinov, Guillaume Sait-Jacques and Martin Tingley

Online experiments measuring whether “A,” usually the current approach, is inferior to “B,” a proposed improvement, have become integral to the product-development cycle, especially at digital enterprises. But often firms make serious mistakes in conducting these... View Details

Keywords: A/B Testing; Experiment Design; Social Networks; Product Development; Performance Improvement; Measurement and Metrics; Social Media

Register to Read

2020
Working Paper

Digital Experimentation and Startup Performance: Evidence from A/B Testing

By: Rembrand Koning, Sharique Hasan and Aaron Chatterji

Recent work argues that experimentation is the appropriate framework for entrepreneurial strategy. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of experimentally testing... View Details

Keywords: Experimentation; A/B Testing; Data-driven Decision-making; Entrepreneurship; Strategy; Business Startups; Information Technology; Performance

SSRN

Read Now

10 Dec 2014
News

Yes, A/B Testing Is Still Necessary

Keywords: statistics; experimentation; testing; products

October 2020
Background Note

The When and How of Effective Beta Testing

By: Robert J. Dolan

Keywords: Beta; Testing

September 2022
Article

Experimentation and Start-up Performance: Evidence from A/B Testing

By: Rembrand Koning, Sharique Hasan and Aaron Chatterji

Recent scholarship has argued that experimentation should be the organizing principle for entrepreneurial strategy. Experimentation leads to organizational learning, which drives improvements in firm performance. We investigate this proposition by exploiting the... View Details

Keywords: Experimentation; A/B Testing; Data-driven Decision-making; Organizational Learning; Entrepreneurship; Strategy; Business Startups; Learning; Performance; Decision Making

Read Now

16 Sep 2024
Video

Standardized Tests

November 1989 (Revised March 1992)
Background Note

Concept Testing

By: Robert J. Dolan

Describes concept testing products. Presents guidelines for effective design, execution, and interpretation of test procedures. Discusses limitations of these techniques and sets out the situations for which they are appropriate. View Details

Keywords: Marketing; Mathematical Methods; Forecasting and Prediction

October 2018 (Revised January 2019)
Case

The Financial Crisis: Timothy Geithner and the Stress Tests

By: Samuel G. Hanson, Robin Greenwood, David Scharfstein and Adi Sunderam

In February and March 2009, the U.S. economy was in the midst of a terrifying financial and economic crisis. Between the beginning of 2008 and early 2009, four of the 25 largest U.S. financial institutions had failed, and nine of these 25 institutions had taken... View Details

Keywords: Bailout; Regulation; Stress Test; Financial Crisis; History; Economy; Policy; Governing Rules, Regulations, and Reforms; Decision Making; Banking Industry; Financial Services Industry; Real Estate Industry; United States

Article

Testing Substitutability

By: John William Hatfield, Nicole Immorlica and Scott Duke Kominers

We provide an algorithm for testing the substitutability of a length-N preference relation over a set of contracts X in time O(|X|3⋅N3). Access to the preference relation is essential for this result: We show that a substitutability-testing algorithm with access only... View Details

Keywords: Substitutability; Matching; Communication Complexity; Preference Elicitation; Marketplace Matching; Communication; Mathematical Methods; Economics

Video

test

04 Sep 2015
HBS Conference

Test Conference

September–October 2017
Article

The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests

By: Ron Kohavi and Stefan Thomke

In the fast-moving digital world, even experts have a hard time assessing new ideas. Case in point: At Bing, a small headline change an employee proposed was deemed a low priority and shelved for months until one engineer decided to do a quick online controlled... View Details

Keywords: Experiments; A/B Testing; Research; Consumer Behavior

Register to Read

Item

Standardized Tests

As you prepare for your PhD application, watch this video to learn more about what standardized tests are required for admission. Topics covered include the Graduate Record Examination (GRE), Graduate Management Admissions View Details

20 Aug 2015
News

Note to Amazon: Run More Tests on Workplace Practices

Keywords: Amazon; data testing; workplace environment; best practices

Article

The Impact of Penalties for Wrong Answers on the Gender Gap in Test Scores

By: Katherine B. Coffman and David Klinowski

Multiple-choice exams play a critical role in university admissions across the world. A key question is whether imposing penalties for wrong answers on these exams deters guessing from women more than men, disadvantaging female test-takers. We consider data from a... View Details

Keywords: Behavioral Economics; Standardized Testing; Gender; Higher Education; Prejudice and Bias

October 2018 (Revised September 2020)
Case

Testing Autonomy in Pittsburgh

By: Mitchell Weiss and Brittany Urick

Pittsburgh’s mayor had been among the first to welcome self-driving vehicles but was now one of many needing to react after a pedestrian fatality involving an autonomous Uber in Arizona. He had originally preferred to roll out “the red carpet” instead of the “red... View Details

September 2011
Article

On Testing Business Models

By: D. Huelsbeck, K. Merchant and Tatiana Sandino

This study explored management decisions regarding formal empirical testing of business models. It documented a test of one company's business model under seemingly favorable conditions for such a test – a successful single product firm following a consistent strategy... View Details

Keywords: Performance Measurement; Non-financial Performance Measures; Business Models; Management Control; Decisions; Business Model; Performance Evaluation

SSRN