Reasoning Chains for Data Analysis

If you're using Claude and ChatGPT to analyze large volumes of data for your business, you might notice that it can come up with conclusions, assumptions, correlations, all sorts of different outputs that lead you to believe that they are correct because, you know, these models are very, very good. However, sometimes you find that they were not correct In that there are hallucinations, there are miscalculations, there are inferences and assumptions that lead your analysis to be partially incorrect or when it's really bad, completely wrong on its face. So in count here, I've asked agent to look at our warehouse, in this case, it's a BigQuery warehouse, and to understand why net new ARR dropped last quarter and to tell me if it has anything to do with this new product that we just released.

So as you can see here, using one of Anthropic's models, it has done quite a bit of arithmetic and has done some research and has made a number of calculations and has concluded that, you know, no, it's not it's not the new product's fault. Really, the ARR shortfall in Q4 is kind of outside of of that rollout. What's really interesting when you begin to dig into Count's reasoning chain are some of the assumptions and issues that the model didn't surface for us that we only catch once we review it. I'm going to focus on this hero answer right here that basically says, hey, don't don't roll back this new product.

If I zoom out, you can see not only does it show the assumptions, the cited numbers, if it had done research on the web, they'd appear here too.

But it also, here at the top right, shows you the core judgments and assumptions that caused the reasoning chain to flag it in the first place. If I click on this one inferred assumption, it takes me to where that node is. And it says in this case, there are two datasets that reflect real exposure during this ARR drop, and we don't actually know the exact dates within the quarter where the product was rolled out or what those efforts looked like. Basically, it's saying we've made a conclusion that seems plausible, but we don't really know.

Right? Like, this this is an assumption. Additionally, if I go over here to the top left, we have a number of different tabs, including the lineage tab, which includes issues, assumptions, and sources. So let's say within issues, I click on this reasoning issue down here.

And it's basically said, hey, we have, as part of this analysis, this Claude model has found that the temporal nature of part of this analysis is off. Basically, it's saying something about the quarter that we're analyzing for when really the data doesn't include the quarter, it just includes when it was flagged last and is making an assumption about that being last quarter. We don't actually know when the the product tag was applied to the data or not. So in this case, I could ask the agent to fix it right here.

I could also ask the agent to fix everything. But in this case, I'll just ask him to fix this one. It sends a summary to it, and it will update the analysis accordingly, which includes, because you saw everything connected, everything up and downstream. So there you have it for data analysts and data teams out there using these models. You know the pitfalls of hallucinations. You know how hard it can be to check for their accuracy. Now with reasoning chains and count, it's a whole lot easier, and we hope you enjoy it.