Article summary with The Discipline of Business Experimentation by Thomke and Manzi - 2014

When it comes to innovation, most managers must operate in a world where they lack sufficient data to inform their decisions, leaving them often to rely on their experience or intuition. Managers can, however, discover whether a new product or business program will succeed by subjecting it to a rigorous test. Yet, many organizations are reluctant to fund proper business experiments and have considerable difficulty executing them.

The Discipline of Experimentation with Business

When it comes to innovation, most managers must operate in a world where they lack sufficient data to inform their decisions, leaving them often to rely on their experience or intuition. Managers can, however, discover whether a new product or business program will succeed by subjecting it to a rigorous test. Yet, many organizations are reluctant to fund proper business experiments and have considerable difficulty executing them.

Most tests of new consumer programs are too informal, as they are not based on proven scientific and statistical methods, leaving executives to misinterpret statistical noise as causation – and making bad decisions.

Companies should conduct experiments if they are the only practical way to answer specific questions about proposed management actions. Also, they should only conduct an experiment if they know exactly what they want to learn. Only then they can decide if testing is the best approach and if so, what the scope of the experiment should be.

Very often executives need to go beyond the direct effects of an initiative and investigate its ancillary effects.

Before conducting any test, stakeholders must agree how they’ll proceed once the test results are in. Experiments are often needed to perform objective assessments of initiatives backed by people with organizational clout. When constructing and implementing a filtering process which decides what experiments will be conducted, it is vital to remember that experiments have to be part of a learning agenda that supports a firm’s organizational priorities.

Experiments must have testable predictions, yet the causal density of the business environment can make it very hard to determine cause-and-effect relationships. Environments are constantly changing, the potential causes of business outcomes are often uncertain or unknown, and so linkages between them are frequently complex and poorly understood.

To deal with environments of high causal density, companies need to consider whether it’s feasible to use a sample large enough to average out the effects of all variables except those being studied. Unfortunately, this is very often not doable.

However, it should also be said that managers sometime mistakenly assume that a larger sample will automatically lead to better data. The required sample size depends in large part on the magnitude of the expected effect.

When deciding on experiments, companies usually have to make trade-offs between reliability, cost, time, and other practical considerations. Three methods to reduce these:

1. Randomized field trials – A large group of individuals with the same characteristics and affliction are randomly divided into two groups. It is closely monitored then whether the treated (test) group does better than the untreated (control) group. Randomization is essential, as it helps to prevent systematic bias. Still, successful randomization can be challenging.
2. Blind tests – The Hawthorne effect is the tendency of participants in a study to modify their behavior when they are aware that they are part of an experiment. Blind tests can minimize this effect.
3. Big data – The majority of consumer transactions occur in channels such as retail stores, where sample sizes are often smaller than 100, violating typical assumptions of many standard statistical methods. In order to minimize the effects of this limitation, firms can utilize specialized algorithms in combination with multiple sets of big data.

Sometimes firms pay a lot of money to conduct an experiment but then do not make the most of them. Therefore, executives need to take into account a proposed initiative’s effect on various customers, markets, and segments and concentrate investments in areas where the potential paybacks are highest.

Another thing companies might use is ‘’value engineering’’, where only the components that have an attractive ROI (return on investment) are implemented. Business experimentations allow companies to look beyond correlation and investigate causality, as sometimes executives only have a fragmentary understanding of their businesses, and the decisions they make can easily backfire.

The most important thing is that a lot of companies are finding out that the actual conducting of an experiment is only the beginning. Values come from analyzing and then exploiting the data.