In the past, I’ve recapped episodes about different metrics including retention and upsell/expansion. But on this post, I want to take a step back and look under the hood of setting metrics.
What’s up, everyone! This recap is from the episode titled “North starts are leading you astray” from the Product to Product podcast by Roadmunk. They sat down with Brian Balfour the CEO of Reforge.
The details
Before starting Reforge, Brian was the VP of Growth over at HubSpot. He’s also started multiple VC backed companies, and grown user bases to millions of daily active users. I participated in a Reforge program back in 2018. The focus was around retention and engagement and I can say that the program taught me more than a year’s worth of courses in university 😅
Product to Product is a podcast about product, for product people! It’s very conversational and candid with a lot of different topics that range from design to machine learning.
So let’s get to those takeaways
Output vs. Input metrics
The constellation metric
Most misleading North star metric
Output vs. Input metrics
Let’s start by saying that, they are NOT the same.
Output metrics are too broad, Brian says.
When you set a metric at a high level for an organization or a team, you’re setting an output metric. But this shouldn’t be used to set goals for your team.
If you tell your team to go and prove weekly/monthly users, there are so many levers that fit under that output metric that it’s really hard for the team to come up with a focused set of ideas and actually move that number.
This output metric needs to be broken down into more granular ideas or input metrics. Levers that if you improve on these smaller input metrics, it will eventually move the output metric. Input metrics give a team a clearer path on the improvements that should be made along the way.
The kicker is that output metrics are always lagging indicators!
So you might pull on a lever and prove it but you might not see that impact immediately on the output metric. This is a huge problem for execs in larger organizations. They’re most commonly looking at the output metrics. So while the team is celebrating the small input win, since the execs aren’t seeing it impact something like revenue, it’s hard for them to see the success.
The constellation metrics
Here’s a hard pill for us all to swallow “there’s never going to be a single metric that really captures all dimensions of your business”.
Teams seem to almost oversimplify and take the concept of defining metrics to the extreme.
But luckily, 3 metrics can give us a full representation of how the product/service is doing.
Breadth of retention
Depth of engagement
Monetization metric
This is your constellation of metrics
Breadth of retention
This is where you and your team want to start. The question here is what is the natural frequency of this product/service?
What is the problem I’m solving for the user and how often does this problem occur in their lives?
Something like Slack, for example, is used daily. Then we go through the process of saying “what is an indication that we are solving this problem for them?” This question is trying to identify the behavior within your product that indicates that you’re solving that problem.
From there, you can focus on what you’re measuring. Is it a user or a team? Of course, this can vary depending on the brand.
With Pinterest instead of weekly active users, you’d measure something like weekly active re-pinners.
The bottom line, retention is a binary metric. A user is either retained or not.
Depth of engagement
To get customers retained, they need to do one or more of some action that you have defined as being an authentic representation that you are solving that problem.
There are a lot of levels of depth of that engagement. Within that time, somebody can take more than one of those actions out of those retained users so you want to know what the spectrum looks like.
The deeper the engagement (the more they are doing that action within that time) the higher the stronger the correlation with longer retention.
Let’s use Pinterest again as our example. With Pinterest, we’re measuring weekly active re-pinners as our retention metric. The engagement metric would be the average number of pins/re-pins per active re-pinner.
You can also break this up a bit so you define different buckets where these customers fall, like casual, core and power users based on the depth of activity.
Monetization metric
Out of the active users, how does this connect to generating dollars for the business?
This metric really depends on your business model. So if you’re in the business of advertising, then you measure this metric by looking at the average revenue per user (ARPU) for that time period you set your retention around.
And for Subscription businesses, you’re focusing on MRR or ARR based on the time period defined in your retention metric.
The full picture
All three perspectives give you the full perspective! You need to focus on all 3 because they impact each other.
For example, you could be increasing retained users but the average depth of engagement could be decreasing. This means that while you have a higher amount of retained users, on average, they are not as engaged. As a result, these users aren’t going to retain for as long. This will also eventually show up in the retention metric down the road.
Most misleading North star metrics
Most misleading North star metrics = Revenue metrics
This is specifically troublesome in the b2b world. It seems like everyone is rallying around monthly recurring revenue (MRR) or annual recurring revenue (ARR) but again the revenue is a massive lagging indication for usage.
What ends up having with b2b products, they have these MRR/ARR numbers but when you look underneath the surface, at the usage numbers, there’s not a lot of product usage. What you’ll see is what Brian calls zombie users––these users will churn at some point!
This begins to create misaligned incentives. Like not wanting to email these zombie users because it’s going to prompt them to churn. Sadly, they’re going to churn anyway so wouldn’t we want to know what the problem is and fix it?
Summary
Don’t let these North star metrics lead you astray:
Output metrics are your goals, you still need to set input metrics to guide your team
3 metrics give you a complete perspective on your business
Just focusing on a revenue metric alone is the biggest mistake we’re making in the B2B world
Shout outs
Thanks to Product to Product for hosting Brian on their podcast! Go follow them on Twitter