Clear your head. Set aside your biases. Put your game face on.
It is time to improve your product.
This is not about sketching a masterpiece from scratch. It is sometimes about finding those hidden blemishes, and at times about adding a splash of color here or there, and sometimes about finding that one thing that translates a simple painting into something of true delight.
However, what it is most importantly about – is about finding the shortest and the most efficient path to taking your product towards success.
I have seen too many product folks; business leaders and even CEOs show an uncanny aversion to experimentation. They are threatened by it. They believe that it is antithetical to their convictions, gut feel and experience.
They can’t be farther from the truth.
Having an innate sense and understanding of what users want, a strong gut feel for possible problems – these are all very useful tools in structuring great experiments. Folks who run experiments scientifically, are not bereft of gut feel or great product understanding. It is just that they have understood that art and science go hand in hand. In fact the boundaries between the two are sometimes rarely visible.
If you like building products, or if you are a budding entrepreneur or an opinionated engineer, or anyone who has a thought on the product – develop the skill of experimentation. Stop shooting in the dark. You will not need to gamble your success away.
Experimentation: The Stages
Stage 1: Build a hypothesis
Use Data that is there at your disposal and dig deep to find or ‘sense’ patterns. Or use your gut feel, past experience and customer interviews to arrive at hypothesis that can be structured. It could be something as basic as a design element, color or text element on a screen being sub optimal or something much more complex with multiple factors at play. Never downplay the importance of research in coming up with hypothesis. Without it, your hypothesis could be just junkyard natives.
State the hypothesis loud and clear.
State the noun, the verb and the metric. State what the implications of the hypothesis being true would be.
Stage 2: Structure the Experiment
A rare, really rare talent. In all my years as a product leader and mentor, I have seen very few product managers with the ability to structure an experiment well. Thankfully, it is an art that can be learnt and codified.
Define the goals of the experiment clearly, and create variations that will test your hypothesis. Use A/B testing or multi-variate testing based on the hypothesis that is being tested, also factoring the time period by which you want to arrive at a conclusion, and the sample size you will be able to run your experiments on.
Plan your variants. Keep a clean control. Absolutely keep clean controls. And of course - no apples and oranges.
Stage 3: Estimate the Effort, Simplify & Prioritize
Given enough time and unlimited resources, you will want to run all sort of fancy experiments. Prioritize what experiments you wish to run based on effort, impact and complexity. Re-evaluate your high effort experiments and see if you can re-design the experiment into a much simpler and lighter form, and not lose the sanctity of the test.
Create and design as many experiments as you wish to. Go crazy with it. But always prioritize. Be disciplined.
Stage 4: Run the Experiment
Kick-start the experiment and drive enough users to sections where the test is running. If you don’t have a sufficient sample size all your inferences would be statistically insignificant. Don’t let your experiments die a painful death by denying it oxygen.
At the same time don’t prematurely turn your experiments off based on a few days of data. Each experiment will need its own cook time – depending on the metric, the environment, confidence levels required, you might have to run them for different periods of time.
Capture data. As much as you can. You never know which bit you will need to revisit.
Stage 5: Analyze the Results & Plan next steps
You are not your hypothesis. Repeat that to yourself.
Your favorite test variant doesn’t need to be a winner for you to feel like a winner. If you have ‘learnt’ from the experiment, then you and the product are both winners.
Look at the data hard, at its significance, clean out the noise and other biases and see what story it tells you. The outcomes could be surprising or as expected – but always double check your conclusions and inferences. Having done that; then again clear your head out and reset your state.
You now have new data and learnings. Use it to decide upon the next course of action – run another experiment, plan a feature or do nothing.
Multiple next steps are possible – weigh each of them and be doubly sure how well linked they are to the results of the experiment and the hypothesis that you just tested. In the urge and fascination to use your new found knowledge; do not jump into a course of action just for the sake of it. Be phlegmatic.
Experimentation done well puts your company and product on to the path of success, much faster than it would normally.
Use it wisely. It could be your superpower!