The dark side of good decision making

Back in 2009 I was working as a junior analyst working for a manufacturing company specialising in enamel coatings. I had just finished my degree in mechanical engineering and (surprisingly, given enamel is exactly as interesting as you’d expect) I was loving applying my statistical knowledge to real-world problems.

For one particular project I was asked to model the implications of switching the supplier for one of the factory’s major raw materials. I modelled the cost and quality implications and presented my findings to the factory management team. My calculations were pretty convincing. Switching supplier would bring a 25% reduction in the cost of product for an almost negligible decrease in quality. The factory manager disagreed. He refused to consider any cost savings which could even risk a reduction in product quality and the project was canned.

This was a pretty painful and formative lesson for me. Having just left university I was brimming with the power of logic and its ability to conquer all. But here my water clad logic was being dismissed, in my view, almost irrationally.

Data Isn't everything

I never did get to the bottom of that decision, but as I’m now the CEO of a data analytics company clearly I still believe in the power of data to make good decisions. The evidence that data improve outcomes is substantial¹ and the huge growth of the analytics industry empirically shows that evidence is convincing. But, as my anecdote above shows (and if you’ve ever been an analyst you will almost certainly have your own example of this), in no way does it guarantee a good outcome.

As an industry we talk about ‘data-driven decision making’ or ‘being data led’ as the pinnacle achievement - the holy grail of how a company can operate. Again empirical evidence shows² that there is definitely truth in this. On aggregate companies who use data successfully do have a huge strategic advantage, but this is also an over simplification. Data can be a hugely important input to a decision-making process, but clearly it isn’tallthat’s required to get to the best decision. People make good decisions all the time despite the absence of good data (even if it’s just luck!) and people with all the data they could ever want can still make bad decisions, so there are clearly other factors at play.

Over the last few years I’ve become increasingly fascinated by the role these other factors have on our ability to make better decisions. Some of this thinking has shaped our data analysis platformCount- for example we strive to make Count the most collaborative data platform in the market because we know discussion and communication is key to help teams make informed decisions quickly.

You can read more about our views on data analysis and our rational behind Count in other posts on our company blogMore than Numbers. But I’m writing this mini-series of posts to explore these ideas in a broader context. I want to take a journey exploring the other, for want of a better term, “softer” factors that impact our ability to make good decisions. I plan to explore what different types of decisions there are, what a good decision-making process looks like and some more philosophical questions like what a “good” decision even is.

My hypothesis is that by having a better understanding of the “art” (or science) of decision-making and exploring all the different variables at play, we’ll find ways to improve the outcomes of important decisions in a potentially equal or greater way than the simple addition of more data-driven evidence.

“But Ollie!”, I hear you cry, “There’s already tons of content about this kind of stuff, why do you feel the need to add to it?”

This is a fair question. There is certainly no shortage of content about or related to making good decisions. My social feed is full of articles like “The top life-hacks of successful entrepreneurs”; “X,YZ study suggests humans make irrational decisions” or “Investment strategies that never fail”.

I’ve read lots of this material (and intentionally avoided many others) and looked for what the go-to books are on this subject and what has surprised me is the lack of attempts to try and rationalise all this material. All the content out there gives a facet of the answer but I can’t find anything out there which attempts to wrestle with all these concepts to unearth the core principles or frameworks that underpin the best way to make great decisions.

Only then can we turn decision-making from a dark art into something we can actually understand and improve at.

Now I’m not going to for a minute, pretend I can achieve this, but I’m certainly keen to explore the topic in more detail and rationalise from it what I can. The aim of the decision making mini-series will be to share my journey and hear and learn from others on their take on this topic.

My general approach will be to explore the problem in three different strands:

Some good old-fashioned musing

I’ll share my pondering on the topic and attempt to distill what I’ve been learning and uncover new questions which I want to learn about which will feed into the other two strands. (This post is a good example of this kind of thing.)

Consolidating available content on a relevant and adjacent topics

I’ll do some good old-fashioned desk research in a particular area (be that psychology, economics or finance) and share back what I’ve found.

Deep dive case studies and interviews with people who have had to make big decisions and how they did it.

To counterbalance all the theory I’ll aim to dig into real-life examples of right decision-making to see what lessons can be learnt and how any theory meets reality.

This is very much a journey of discovery for me and I’m not really sure where we’ll end up. But I’d be very pleased to have people along for the ride with me. In particular I’d like to hear if my general bemusement about this topic resonates with you and/or if you have any good resources which you think should steer my early research.

Until next time...

¹ I could insert about 100 white papers here talking about the benefits of data, just google “Power of data” and fill your boots. But assuming you’re sick of white papers here’sa comic about data instead.

² Seriously, just google it...