Experimentation encourages innovation, but it can also be time and resource-draining. To make experimentation a productive activity in your organization, you must manage several conditions, including the key considerations one must make when testing in an ever-changing business environment, which data to test against, and the criteria against which one should make decisions
Understand testing in business conditions
You should consider the opportunity costs inherent in testing and be willing to adjust methodology and scope accordingly.
- Managerially actionable testing can still yield insights and options. The goal is to generate relevant dialogue among decision-makers in changing market conditions, not eternal truths.
Pay attention to “small” ideas
Companies tend to allocate testing time and money to big initiatives while ignoring small ideas that, in the aggregate, can have a bigger impact with less risk.
- Seek progress, not perfection, and invest in processes that allow employees to submit seemingly small ideas.
Mind your data
You need reliable data to avoid the garbage-in-garbage-out syndrome
- Common examples involve tests driven by data in customer relationship management (CRM) systems
- The inputs are noisy because the system reports the aggregate result of what, in reality, is multiple people using different criteria
- Once the system is in place, however, tests are then designed to optimize for the software parameters, reinforcing an outdated view of consumer behavior
- Churn rates can linger for years
Establish decision-making criteria
Managers must always interpret data with an end in mind
- Price testing should be an ongoing part of effective marketing, but first clarify the evaluation criteria because testing in business ultimately means evaluating alternatives
- There’s a big difference between using profit increase or revenue lift as the criterion and price changes that have an impact over multiple time periods, not just in the short term
