WTF is a Metric? – introduction to the basics of metrics, what it is, and why it matters. Part 2: 4 Steps to Defining GREAT Metrics for ANY Product – how to develop a metric from scratch and avoid the unintended consequences of the “one metric that matters.”
Beyond this series…
Metrics matter
- Defining metrics is as intrinsic to the product development process as defining requirements
- One of the most frequently asked product manager interview questions is: How would you define the success metric for feature X?
- The answer to this question is: Metrics should be part of the process
Game: Facebook’s Newsfeed Example
Goals for the Newsfeed can be broken down into user and business goals
- User Goal: I want to be able to see the latest of what my friends are up to without going to each of their profiles
- Business Goal: Users are only coming to Facebook when they have a reason to
- Direct vs. Proxy: Most of the actions above can be tracked directly with clicks except for viewing a feed item
- Individual vs. Aggregate: Because our primary goal is measuring engagement, we care about our users engaging with the newsfeed in general more than any specific action, so we will define a metric with aggregate measures
- Magnitudes vs. Ratios: Looking at the magnitude of Newsfeed action is relatively meaningless since any given user can generate a great number of actions
- Intrinsic vs. Heuristic: Use this data to develop a heuristic around an engagement metric, whereby if a user crosses any of the three thresholds above, they are considered “engaged.”
- Pulling this all together, our metric for the Facebook Newsfeed will look something like this…:…where “engaged user” is developed by aggregating both direct and proxy actions and comparing it as a ratio to “total users” on a daily basis
The Challenge
Developing metrics from scratch can be incredibly daunting, precisely because there is so much at stake
- Not all data is created equal
- Some data are more valuable than others because they give you a unique insight into your product’s health
- There is a lot of subpar data that can cloud your judgement
Define Your Metrics
Major decisions for how to count each action
- Direct vs. Proxy
- Individual vs. Aggregate
- Magnitude vs. Ratio
- Intrinsic vs. Heuristic
- Can you derive more knowledge from the measure intrinsically? Or do you have to rely on heuristics for the metric to be valuable?
A Mental Model for Metrics
In order to build a product your users value, you first need to determine the inputs and outputs of this box.
- Inputs are anything your users come to you with
- Outputs are the circumstances resulting from the use of your product
- Most products require users to take actions within the product to extract value and achieve their goals
- Quantifying and measuring actions gives you insights into how many users are getting value from your product
Step 4: Evaluate your metrics
Is the data trending in the way you expected? Is the metric stable over time? Are these the correct data points to collect?
- Evaluating Actions
- Are these actions reflective of product, user, or business goals? Are your key actions telling the correct story about user behavior? Does your action set cover all emerging user behaviors? Is your actions safe from being “gamed” by your users? If you answer “no” to all of these questions, you want to redefine the set of key actions
- Revaluate Goals
- Look for metrics to measure vital signs that give you an actionable indication of health and performance
Defining Your Metrics Before You Need Them
Having metrics defined before you start development can translate to huge benefits
- Help maintain focus
- Set a common goal for team to rally around
- Hold you accountable to your users and business
- Enable more objective decision-making during the critical launch period
- Flying blind at any stage of your product’s lifecycle will substantially decrease your ability to learn
Step 1: Articulate Your Goals
Define your goals for your product.
- User Goals: How will my users benefit from my product?
- What problems do my users want my product to solve for them
- Users interactions with the product
- How will I feel when I use the product? What is my vision for how this product will integrate in my users’ life
- Business Goals: What are the tactical or strategic business benefits my product will bring me?
List the Actions That Matter
Define user actions, or more precisely, all the actions you want your users to take within your product.
- The key to doing this well is to be comprehensive but not exhaustive: have the broadest coverage, but don’t bother differentiating the minutiae.
