Count AI Beta is here!
[Quick recap] Two weeks ago we chose a nice light topic to cover for our first article: What is the purpose of a data team? We assembled advice and approaches from many data leaders to create not a single over-arching purpose, but a framework by which we can see and understand the spectrum of data teams we see today. These three data team roles were: enablers, advisors, and connectors.
While I believe the framework is helpful, it’s important that this newsletter shares immediately applicable tips and tricks in addition to theory - things that you and your team can implement tomorrow and start seeing results right away. We will call these playbooks - and this week is our first, and it’s all about stakeholder surveys.
Big thank you to the many folks who contributed to this article, but an extra big thanks to the following for being kind enough to share their stories and surveys!
Measuring a data team is difficult, for reasons I will discuss in depth in an upcoming article, but the biggest challenge is that so much of what we do is not quantifiable. For example, the utilization of a dashboard doesn’t tell you if someone trusted the numbers they saw, made a decision based on that, and if that decision was a good one or not.
“I’ve come to a realization that the data team, even though we’re driving things forward and we’re trying to get deeper insights and drive deeper strategic impact, the reality is we’re still a service organization. And the best way to assess a service organization is to have some sort of NPS figure. But the NPS doesn’t have to be all promotors and detractors. It can really focus on the quality and availablity of the data and how its used by stakeholders.” - Lina Mikolajczyk, Director of Advanced Analytics at Bumble
Enter stakeholder surveys.
Stakeholder surveys are designed to measure all of those intangibles that reveal how we are really doing. How is the business really using data? Do they feel confident? Do they really understand their metrics or just get tired of asking you what everything means? What do they wish you were doing more or less of?
And with this information, we can make sure we’re working on the right things and doing our best to support our teams.
In another role before Bumble, Lina was Head of Data Science and Analytics for another company. When she arrived, she joined a team in flux. Based on some company-wide internal surveys her new data team:
It was clear there were some serious problems, and to get to the bottom of it, they had to go out and talk to people. In addition to one-on-one and small focus groups, they also set out to do a company-wide survey to find out what people thought:
From this simple survey, they learned:
With the realization that neither the organization nor her team were very happy, Lina took action. She worked with department leaders, managers, and her team to process the survey results and start making a plan to correct the major issues it uncovered. The changes included:
Lina and her team were able to dramatically reduce attrition on the data team, and the data team's trust scores shot up to the company average.
Furthermore, when Lina started at Bumble, one of the first things she did was implement another company-wide survey to take stock and make her team’s roadmap.
Don’t just put out a survey for the sake of it. What do you want to get from this? Some examples are:
For the survey to be helpful you need people to use it. To do that, you need to get the word out any way you can.
“I went into the organization’s all hands and said this is who we are, this is what we do, this is where we play. We’re asking you do do this survey because we want to assess whether the services we provide you are what you need and whether we’re doing the most impactful things because we want to build a proper roadmap for next year based on your results.” - Lina Mikolajczyk
Some other suggestions from Lina and Daniel are:
No more than 4-5 questions. You shouldn’t need more than that if you are being razor-focused.
While surveys should be anonymous, you can ask questions to help work out different populations of respondents, such as:
Include open-text questions where people can leave their thoughts and explain previous responses.
“I’m always surprised at how much people are willing to share when you ask. I didn’t expect anyone to use the open text field but nearly everyone did, and it became the richest source for feedback we had.” - Lina Mikolajczyk
Daniel takes this one step further by allowing people to chat more directly in the survey. His final question asks if they would be willing to talk about any of their responses and if so, to offer their name so he can message them directly.
We talked to Lina, Dan, Phillip, and others about questions they ask in their stakeholder surveys and assembled a list. This is a good starting point if you’re looking for inspiration:
See example stakeholder survey questions
Let your leaders know what you’ve found, and have a discussion about what you plan to do about it. Often the survey may reveal hypotheses you’ve already had about the problems facing your team and the larger organization around data, but the survey can help turn those into quantifiable facts. With their support, you and your team are more likely to be successful.
Working through the results with your team can be a painful process. Many data teams don’t struggle for lack of effort. We have all poured our hearts out on data teams trying to be as helpful and effective as possible but never seeing the fruits of that labor.
Be sensitive to how poor survey results may demoralize your team, and be sure to focus on what we all stand to gain by changing as a result of these findings. This isn’t a punishment, it's an opportunity for us all to have a better time.
“It’s important that you follow up with the organization so they know it was worth their time to do that survey. We immediately made the results available in Slack as well as what we were planning on doing with it. When we run data trainings for new hires now, this is the dataset we use as well. It’s become a shared artefact, not just something for the data team.” - Daniel McNamara
Being transparent with the organization about the results and plans to correct any issues you’ve found is key to establishing trust and holding yourself accountable for better results in the future.
Continue to run surveys regularly and use them as part of your roadmap, OKRs, and other measurement frameworks.
Next week we’re diving into the abyss of measuring a data team. The topic has been a rollercoaster to dive into and I’m looking forward to sharing the journey.
