Count AI Beta is here!
After asking data leaders what the purpose of a data team was, I asked one follow-up question: ‘How do you know if you’re successful?’
This was almost universally followed by a heavy sigh. This is not a simple question.
We in data have never been great at measuring ourselves. Despite our role in helping others measure their success, we lack a standard set of metrics, or even a framework by which to measure our own.
“You’d think since we’re in the business of measuring things, we’d be great at measuring ourselves, but we’re not. We’re probably worse at measuring oursleves than we are anything else.”
- Conor O’Kane, Head of Analytics at Cleo AI
Articles on the topic generally gravitate towards laundry lists of operational metrics like dashboard utilization and time to insight. But do those really tell us if we’re doing our jobs well? (They don’t).
You won’t be surprised to hear then that this article was particularly challenging to write, but one I thought was well worth the effort.
The following is everything I learned from these leaders about the different approaches to measurement, the patterns I found, and the advice they offered for those on the daring mission to define and measure their own success. Here it goes…
Thanks to
for your help navigating this heavy topic!
Of all the data teams I’ve known and worked with only about 10% of them had formalized metrics for themselves. Many of these teams without any kind of measurement were still objectively healthy - they were valued in their organizations, and they seemed to have satisfied team members. They just didn’t have any way of quantifying all that, which begs the question, do we even need to measure ourselves?
The simple answer? No.
In fact, measuring for measuring sake can be harmful:
“If you’re in an environment where you need to really, really distinctly show the impact of a data team or an analytics team, your data and analytics team will end up working differently. That’s because they have to work in a siloed way so that you can draw a dotted line around them and isolate the effect they’re having. In trying to make our impact easier to measure - we in turn lessen the impact we can have.”
- Alan Cruickshank, Insights Director at tails.com
But would any of us even be in data if we didn’t understand the potential value in measurement, especially when it is done well?
Measurement done right can be an invaluable activity - one that helps us understand ourselves and our wider context better, and that can help us deliver more value to the wider organization.
Teams that did have some kind of formalized measurement framework did so for the following reasons:
We’ll dive into these different motivations for measurement below. But first, a brief digression.
One mistake many people highlighted was using metrics to help prove your team should exist.
When leadership asks you for metrics to quantify the value of the data team, the question behind the question is usually ‘Do I really need a data team?’ or ‘Do I need one of this size?’. Those concerns won’t be quelled with your utilization numbers or your average time to insight.
And in case it wasn’t obvious, the costs of not addressing their concerns can be high:
“Over the course of my career I have seen multiple data science teams who were doing fantastic work in building and sharing models getting dismantled overnight. The simple reason was that the business wasn't seeing their value. The value was there. I could see it. But the business didn’t.”
- George Spiteri, Head of BI & Analytics at Chilliz & Socios.com
So what should you do if you’re asked to ‘prove your value?’ Here are a few tips from the leaders I spoke to:
As a very frustrated HR colleague once told me “Frustrations are often just a failure to communicate expectations.”
Meaning the reason you’re being asked to demonstrate your team’s value is likely that no one knows what to expect of you. Or they have expectations of you that they haven’t told you about, so they perceive you to be failing.
For example, if you believe your data team’s role is to advise the business on good decisions, then someone can be very frustrated that you aren’t just serving them reports and numbers as quickly as possible. You believe you’re partners, and they believe you’re a service team.
So, one of the most powerful ways to address the existential question is to clarify who you are, and what others should expect from you and your team. This practice of consistently and proactively communicating who you are, how you can help people, and what people should expect of you is one of the things I’ve noticed can distinguish good from great data teams.
If you’re looking to reset the expectations of your data team, try the following:
And if you haven’t check out our earlier article on finding your team’s purpose here.
If you still find yourself trying to justify how valuable you are, it can be helpful to focus on how other people in the business demonstrate that they value you.
💡 For example, one team I spoke to has relinquished their data budget and given it back to the larger company. They then ask for business units to pay for their own analytical resource. This is perhaps the best sense of how much they value the role of the data team since they are putting a dollar value on it, and some of the best advice I received on this topic.
Some other examples of demonstrated value are:
In my conversations, I observed that data teams with embedded analysts were asked to defend their existence far less often than centralized teams. The organizational structure creates a shared sense of value that is so often felt that it is rarely called into question:
“What I like to do is make us more visible and integrated into the business teams to show the value we bring. I try to have every analyst a part of the projects they work on, not just supporting with numbers, but participaing in the decision-making and project management. It’s not quantitative but does create a valued image of Data Analysts among the business teams.”
- Margot Monges, Data Analytics and Data Engineering Manager at Dott
In short, the question that seems to be all about metrics is probably not about metrics at all. But now, back to the regularly scheduled programming.
Setting targets and goals for the team is an incredibly useful exercise. These goals help unite the team around a common mission, and ensure that everyone is working towards that outcome. This can take a few different shapes though.
As we mentioned above, most teams know if they’re successful if the business is also successful. However, it’s helpful to break that down further. What does it look like inside the data team when we’ve achieved that outcome?
These more internal objectives help focus the team on what’s important:
“[To know if we’re successful] I look at the strategy docs that are coming out of each department and seeing how aligned those are with the insights coming from our work. Sometimes I’ve had the analyst actually writing those memos. That means we have a seat in the room and are making a difference.”
- Triston Cosette, Principal Analyst at GetYourGuide
Other data team goals I’ve heard are:
Many times these goals can help influence behavioral changes within your team. For example, Carl Johan Rising, the Director of Data at Too Good To Go, is planning to experiment with this idea in the New Year by introducing a new key metric for his team: the percentage of work tied to decision-making.
“They need to ensure that they always ask that question - what decision are you trying to make, and by what criteria? This goal will help encourage that behavior.”
- Carl Johan Rising, Director of Data at Too Good To Go
Carl Johan’s plan underscores the purpose of setting good goals in the first place: to focus a group of people around a shared ideal outcome. With that idea, your team goal can be as broad or as narrow as you like. The power comes in simply choosing one.
Ah, yes, everyone’s favorite thing to measure. This group includes metrics of efficiency, downtime, coverage, utilization, satisfaction, and speed. Ideally, they help us see whether we’re on track to hit the goals we’ve set above, so they should reveal two things:
Most data teams are good at capturing these kinds of metrics, which can be a weakness. Many teams I spoke to had at least one experience of tracking too many operational metrics or simply focusing on the wrong ones.
Therefore the key with these metrics is to be selective. For example, don’t worry about freshness until you find out that it is causing trust issues, which is diminishing your ability to influence the larger strategy.
Other examples of operational metrics include:
It’s also important to balance operational metrics with qualitative feedback from stakeholders, such as:
Having a set of goals for your team that are about job satisfaction can be a powerful way to build morale and keep the team motivated.
Anders Kring, Head of Data at Templafy, has the following things that he wants his team to check for themselves every day:
He reviews this with his team regularly as a group and individually so they can calibrate their priorities and ensure they’re having a good time while delivering value for everyone else.
Looking at all of this can be overwhelming, especially if you are one of the majority of teams with minimal measurement in place today. To get started, here are a few questions to ask yourself:
Here’s a summary of the above information:
As always, if you have any topics you want us to cover or any folks you’d love to have us interview, drop me an email: [email protected]. Otherwise, we’ll be bringing you more content in the New Year on topics like hiring, data team structures, and what we do about analysts. There will also be more practical playbooks from teams at Trainual, Substack, Gelato, and more!
