Metric Trees - Advice and Best Practice

Advice from 3 Metric Tree experts.

In Part 3 of our 4-part webinar series Driving Operational Clarity - The Metric Tree Masterclass, I was joined by 3 experienced data leaders to have them share their experience and wisdom around building metric trees. In this session we cover: 

• Advice for getting buy-in and approval
• Lots of tips for breaking down your metrics
• What effect can metric trees have on your organization

Meet our panel:

• Callum Ballard, Analytics Director at Omaze
• Michael Rogers, Associate Director of Product Analytics at Bumble
• William Mahmood, Head of Data Science at MUBI.

Watch the full conversation:

Or get the summary:

Key Takeaways

1. How did you first come across metric trees? What was their initial appeal?
   1. Both Callum and Michael first heard about metric trees from their consulting days. There they often used issue trees to break down complex problems, and metric trees were a natural extension of that. They simultaneously helped communicate the problem to clients, and also propose and motivate the best solutions.
   2. For Michael metric trees have a lot of overlap with growth models and business equations - they both decompose a system and show its inner workings.
   3. Will first used metric trees at a start-up early in his career where he needed a way to show complex metric interactions without data overload.
2. Where should someone begin when making a metric tree? How do you get approval to even begin?
   1. In Will’s most recent experience at MUBI, the metric tree started when the leadership team asked him to help overhaul their Northstar metrics. This was beneficial in that it was very focused and had immediate buy-in from executives.
      1. However, he still had to get a lot of buy-in and support from business leaders when building out the tree. For this, he suggests making the tree really relevant to stakeholders - find out what metrics they care about and show those. And when you do that for many teams at once, you suddenly have something really powerful with a lot of cross-function connections.
      2. Callum echos this by encouraging you to think about what each department really wants. If a product team wants to know how their new features affect a high-level company metric, use a tree to demonstrate that and they’ll definitely want it.
   2. Michael and Callum haven’t always approached it as an official project that requires approval. Instead, they saw an opportunity to change how they were discussing metrics with business users, so would come in with a metric tree and see how it went.
      1. “People are not going to ask for it. Find a common use case where it can be brought up naturally and people can understand it like a weekly trading meeting, or a product review. In these forums people were already talking about metrics, but never how they link together.” - Michael Rogers
   3. Getting buy-in and support isn’t something you just do at the start. Will highlights the importance of being open and working collaboratively with stakeholders throughout the project to build trust
3. How to begin mapping out and defining your tree?
   1. Start with a very simple number (e.g. users, dollars, etc.) and break that down into it’s components.
      1. It’s also important to know the purpose of the tree. This is how you’ll know if it’s working or not as you build it out.
      2. Sometimes you can also take a list of metrics that you know need to fit together and build your tree around those - think of them like pieces of a jigsaw puzzle and it’s up to you to fit the picture together.
      3. Or sometimes for product metric trees, it can be better to work bottom-up
   2. For decomposing metrics, think about the following:
      1. Do your best to make sure everything adds up to 100%
      2. Do your best to stick to achieving MECE (Mutually Exclusive Collectively Exhaustive)
      3. Think about how a metric would break down in terms of growth accounting (e.g. Revenue breaks down into Retained Revenue, Churned Revenue, etc.)
      4. Try to avoid proxy metrics and abstractions
      5. When trying to break down a ratio, use inverse ratios to your advantage: A/B = C/A * C/B (e.g. when breaking total revenue per user, you could introduce the total number of sales as your third element. Total Revenue per Customer becomes Revenue per sale * Sales per Customer)
      6. It’s important that your decomposed metrics are business-relevant. It’s useless unless it means something to the business
      7. Don’t be afraid to use dotted lines when there are things that don’t necessarily mathematically contribute to a metric but are related (e.g. thinking about how the NPS score drives retention)
      8. Don’t be afraid to break away from a strict tree format where it makes sense (e.g. showing funnels or segments within the tree structure)
      9. Don’t over-normalize metrics - that can hide important trends you want to surface.
   3. It’s going to be an interactive process. Don’t get discouraged. Start with a high-level conceptual tree with sticky notes and work into more and more detail. It’s going to take a lot of practice before you get it right.
4. What impact do metric trees have?
   1. Metric trees drive operational clarity - they make the complex business seem simple and clear.
   2. They simplify decision-making. The data can be really confusing but it helps show what’s important.
   3. They help bring clarity and focus to an organization - e.g. helping MUBI focus on engagement over other Northstar metrics like revenue or memberships. There’s a real organizational shift they can lead.

‍

‍

References

• Issue trees: https://www.craftingcases.com/issue-tree-guide/
• Victor Chang: https://caseinterview.com/
• MECE Principle: https://en.wikipedia.org/wiki/MECE_principle
• Proxy Metrics: https://centerline.net/blog/vanity-metrics-versus-proxy-metrics#:~:text=Proxy%20metrics%20are%20the%20formative,your%20audience%20to%20an%20action.