Data teams are busier than ever. Each one I speak to tells me about their latest initiative—whether it's reducing Snowflake spend, implementing new tools, or embarking on an epic quest to build their company's first metric tree.
I have the privilege of speaking with many data teams, and I'm consistently impressed by their energy and dedication to improving their work and helping their businesses.
But despite all this effort, I'm increasingly concerned that this work isn't cutting through where it truly matters.
One of the things that really shocked me recently was a Harvard Business Review survey of Fortune 1000 business that said 59.5% of executives reported their companies were driving innovation with data, compared with 59.5% four years ago.
That’s zero change in four years.
In that time, we’ve seen teams radically change their data stacks, invest in their teams, growing pipelines and infrastructure and this survey suggests the impact of all of that has been zero.
And if we’re honest I think we all feel it.
This inability to measure our impact or feel like you’re contributing to business value is tangible. In dbt’s 2024 State of Analytics Engineering Survey, only 20% of respondents could measure themselves against financial goals or even qualitative evaluations.
Most of us have accepted the only way we can measure success is the completion of work and how we make other teams move. We can’t directly measure of prove our value.
This is not where we want to be as an industry - a place where we aren’t seeing the efforts of all our labor.
Another trend I’ve always been very anxious about is the role of the analysts and how we value analysts as an industry.
When talking about translating data assets into business value, there is no role more significant than the analyst. They are the ones most capable of making a difference.
But as an industry being an analyst doesn’t mean to be the hero of the data stack.
An analysis of data career progressions done by Motif recently found that most analysts move into more technical disciplines and further away from the business, and only 3% move up into more senior roles.
Part of that is driven by the market. In a 2022 StackOverflow survey we found that data analysts make 9% less than their more technical colleagues, even if you take into account their coding experience.
This is just another way that we as an industry are moving away from investing in business impact.
Lastly, I’ve noticed a pattern in the operating models of data teams that’s exasperating both of these previous trends. I call it the service trap.
The service trap is a pattern of behaviours, that in isolation are damaging to data teams, but together combine into a negative feedback loop that is very difficult to break free of.
These four forces are:
When caught in this loop, data teams become “dashboard factories,” valued only for accuracy and speed. Their interactions with the business become transactional and strained.
And most importantly, their contributions to broader company goals are invisible, and their impact on business growth goes unrecognised.
This is how we get data teams investing heavily in maximising efficiency while paying little to no attention to value creation. This is how we get teams that under-invest in analysts and over-index for technical skills.
Continuing down this path can lead to some detrimental consequences for data teams: continued irrelevance in their organisations, low levels of data team satisfaction and engagement, and ultimately less and less resource from the organisation.
Data tools aren’t the reason we’re in this trap - but they aren’t helping us either.
In the last 10 years we've seen a flurry of new tools, but the overwhelming majority of these affect our back-end systems. The way we provide data to the business—through BI tools like Looker and Tableau—remains basically unchanged.
Therefore, the problems we had in 2014 still exist today, just with more robust pipelines and slightly faster querying engines.
These BI tools perpetuate the idea that data teams’ core value is to build dashboards, charts, and now data models, as quickly as possible. They outsource insight generation to the business and give them some drag-and-drop interface and hope for the best. The only ‘new’ feature is connecting to dbt to build those dashboards and charts even faster.
It's clear that the way we're working is missing something fundamental. In dbt's 2024 State of Analytics Engineering, over 50% of respondents said increasing trust in data and data teams was a core objective for 2024. This was the most popular focus area for respondents, beating out shipping projects faster and reducing cost of insights. If we have invested so much time and money into these data stacks and they still aren't curating a culture of trust then we are clearly doing something wrong.
What I find most disheartening about all of this is that rarely do I hear a better idea - a better articulation of what a data team could be doing besides building dashboards, enabling self-service, and answering ad-hoc requests as quickly as possible. Even the latest trend of AI solutions largely just make it more scalable to work in the service trap rather than fundamentally changing it.
We have the best stacks we’ve ever had, we have our hands on the most powerful technologies, and we have the most talented, technically competent data teams we’ve ever had.
No one gets into data because they want to be dashboard factories, or even to build models in complete isolation from business value. These aren’t easy patterns to break, especially with our current suite of tools.
Over the last few years I have come across several teams that seem to have figured this out. They’ve broken free of the service trap and as a result, operate very differently to the typical data team.
On these teams:
While this way of working is the minority today, I’m convicted is the way all data teams should operate. And this vision drives our work every day at Count.
If you like the idea of working in a more meaningful and powerful way, consider this your rallying call. Join us as we do all we can to make this future a reality.
