Count AI Beta is here!
Hello & welcome back from our (long) holiday break. Hope everyone is having an excellent start to the New Year and is ready to get back into our weekly cadence!
Hiring is notoriously difficult, something I didn’t fully appreciate until I was in charge of the process for the first time. And while we aren’t in the hiring frenzy we were in a few years ago, many teams I speak to are still growing, making this topic as relevant as ever.
Last year I spent a few weeks building The Ultimate Guide to Hiring Your Data Team - a (near) comprehensive guide to data hiring, covering pay scales, role definitions, sources for finding talent, types of assessments, and a long list of example questions to use. This time, though, I wanted to focus on the things you can only learn from experience.
The following are the best pieces of advice on hiring I’ve heard across all the conversations I’ve had as part of this series.
Big thanks to the following folks for their help on this article!
“A wrong hire doesn't equal ‘Oh, you hired someone with a low IQ who has no work ethic and can't talk to people’. It's a matter of whether the person has the skills to fit your needs in that particular moment plus the what you’ll need to scale.” - Bill Barron, Staff Data Analyst at Thirty Madison
The biggest mistake you can make when hiring is not being sure what you need. It’s very easy to say you want to find the very best Analytics Engineer you can find, and it’s also very easy to assume “best” means the one with the most qualifications.
Not properly defining what you need at the start of your hiring process can lead to finding some very qualified people who are not prepared for the job you need them to do.
Before doing any kind of outreach, you and your team should be clear on exactly the role you’re hiring for, and ideally what a successful candidate will look like.
For better or worse, the hiring process today is dominated by keyword searches and algorithms doing their best to match candidates’ CVs with job descriptions. This means the words we use can have significant impacts on the candidates we attract.
Jessica Franks, Head of Data at Not On The High Street, found out the hard way that a few words on your job description can have a massive impact on the applications you get:
“Our job description for a Data Engineering role contained a line mentioning supporting machine learning projects, and for a number of candidates we interviewed, this was the only thing they focussed on. Since this was a data engineering role primarily focussed on ELT, we had to rework the language in our job description to remove machine learning as a keyword so we could manage candidate expectations better” - Jessica Franks, Data & Analytics Engineering Manager at Not On The High Street
The takeaway here is not so much that people love talking about machine learning (although that might also be true), but that it is often worthwhile to change the wording in your job description when you aren’t getting the candidates you want. Or maybe circulate several variations of the same job description and see what kind of candidate each attracts (thanks to Alan Cruickshank for that tip).
The value of word choice extends beyond job descriptions though. This can also be the case when looking at your data team branding.
“Adopt modern but appropriate branding for your data team. This has clear effects in the hiring funnel whether you're using recruiters or external job ads. The terms used by a team says a lot about its existing mandate and aspirations. This can be challenging in today's environment given the plethora of titles and lack of standardisation in some domains but is important to get right.” - William Mahmood, Head of Data Science at MUBI
Will and his team have many reasons for being a data science team vs. data analytics, business intelligence, or one of the many other names you might come across. What Will does not advocate for is changing your team name for the wrong reasons. Don’t call your analysts data scientists because you want someone who can make predictive models - only make sure your name matches who you are and the work you do.
While getting the job application (and your team name) right can make a big difference in finding the right candidates, sometimes that’s not enough. Many people I’ve spoken to have highlighted the need to work with recruiters to hire for specialized roles like Data Engineers and specialized analysis (e.g. Marketing Analysts).
“We’re in a complicated domain with highly specialised and varied skills so finding a recruiter with the right industry knowledge and contacts to align with your business is crucial. I’ve worked with recruiters who’ve adopted the “spray and pray” approach after asking for a job spec and a few high level questions and I’ve worked with others who’ve sent over a handful of high quality candidates after taking the time to ask the right questions up front. I know which approach I prefer!” - Tim Potter, Principal Analyst at carwow
If you can afford it, experimenting with specialized data recruiting agencies can be a very worthwhile exercise.
Take-home assessments have been the dominant way we assess someone's technical abilities. However, Matt Hawkins is advising everyone to think twice before doling out another take-home assessment for your next hire:
“My one piece of advice: don't give people take home tests. They’re unfair in that they place an unreasonable demand on a candidate’s time, and risk excluding good candidates who have competing demands on their free time (like kids). Even if you say “we only expect you to spend an hour or two on this,” the candidate will spend the whole week on it — if they can — because they know they’re competing with other candidates. More than that, though, it’s an ineffective way of assessing a candidate’s problem-solving skills, since you get no visibility of their thought process in the moment when faced with an unfamiliar technical or conceptual problem. This is a more useful data point than whether they got the right answer or wrote perfect code.” - Matt Hawkins, Head of Data at Runa
Many candidates had things they suggested to try instead of the take-home assessment:
1) Do a live code review
We put together a case study, which was a few kind of personality-based questions, a few technical-based questions on how would you set up this problem, how would you deal with this thing? And then we gave them an airflow task, because he had airflow experience, the system in question, and said, here's a piece of airflow code. Look through, tell me what it does, what's good, what's bad, what's missing. It also saved them a lot of time, because they don't have to spend six hours preparing for an interview that they have to fit into evenings or mornings. - Bob Clark, Director of BI at Zapp
2) Try a live coding session
“I think [a good alternative] is going through a live coding exercise where you ask them to talk through what they're doing and encourage them to look things up (on their shared screen) when they’re stuck. That way you can see how they formulate a question as a coding problem and assess their Google-fu, regardless of whether they know the right function off-the-bat. Live exercises put problem-solving skills in the spotlight, while placing equal demands on both parties’ time and sending the message that the hiring team is just as invested in the process as the candidate.” - Matt Hawkins
3) Talk about concepts
“Most of the time, I just have a genuine conversation with them and try to dig deeper into technical topics. When they start talking about that, I ask more and more questions on the same topic until we hit the bottom. Basically, we hit the limit where their knowledge is.” - Yordan Ivanov, Head of Data and Analytics Engineering at Dext
4) Combine them for a live technical session
Increasingly I’m seeing teams change the format of their technical interviews to include some combination of discussion of technical topics and concepts, doing live code review and interpretation, and doing some kind of problem-solving task with live code to assess their critical thinking skills.
The collaborative and flexible nature of the Count canvas makes it a good option for running these live technical sessions. In the video below Ollie walks through an how teams at Cleo and Not On The High Street use the canvas for live technical interviews:
5) Don’t skip the behavioral questions.
Sometimes the behavioral interview can take a backseat to the technical assessments in our field, but a few folks highlighted how essential this step is.
“For us, what always comes up on employee surveys is that people love how flexible it is to work here, and they really like the people. So hiring people that are going to keep that going is important.” - Omer Sami, Head of Product Analytics at Vio.com
“Don't let the person just talk about stuff. Sometimes, I will cut an answer every one minute, or I will stop them and ask them to either talk about something else or go back to what they said before. You have to keep things moving so you can get the information you need. There will be time for a free-flowing conversation later but it’s your responsibility to surface the necessary information to be able to make a decision about the candidate..” - Omer Sami
“One question I always ask is ‘What did you do this weekend?’ It might sound trivial but we’re a small team and personal connections are important. You show that you care who the person is behind the employee. Creating belonging, safety and purpose are key for successful teams. As a bonus a lot of time talking about your passions helps people feel more comfortable and gets rid of the nerves.” - Jurien Groot, Head of Analytics at Hubs
Many interviewees acknowledged how this process is still prone to human bias, even if you do everything perfectly. Cleo AI attempts to mitigate the risk of bias by involving many people in the hiring process.
“We have 6 steps in our hiring process, and each one is covered by 1-2 people from different departments. That means by the end, at least 10 people have gotten to know the candidate and vice versa. This enables us to not form biases on the candidate during the process, and also lets candidates get a feel of the culture and the people at Cleo.” - Bianca Nicolas, Senior Business Intelligence Analyst at Cleo
A lot of talk so far has been about how we as the hiring team select the right person, but another huge part of this process is making sure the right people select you too.
The best examples I’ve seen of this look like the following:
Oh so much. We have some playbooks in the works on how to shift out of being a service team, a deep dive into data cost attribution, and maybe even some experiments with new formats…
This newsletter is brought to you by the team at Count. Count is a data whiteboard making analytics and data modeling collaborative, transparent, and trusted. Learn more here.
.png)
