You’ve built an agent. Now let’s make it great.

Your agent is in the wild. Real users are asking real questions. Some are getting good answers. Some aren’t. How do you know which is which - and how do you systematically make it better?

This session is about the operational reality of running a self-serve agent at scale. Alice and Jason walk through how to monitor agent quality using telemetry data, spot the patterns that signal something’s wrong, and update context, schema, and data models based on real usage rather than guesswork.

Most teams’ agents quietly degrade over the first few months because no one’s watching the right signals. By the end of this session, you’ll know exactly what to watch and what to do about it.

This is Part 2 of a three-part series. It builds naturally on Part 1’s ‘how to build it’ - but stands alone if your agent is already live.

Frequently asked questions

• I haven’t watched Part 1 - can I still get value from this? Absolutely. Part 2 stands alone. We’ll briefly recap the building blocks at the start so no one’s lost.
• Do I need an agent already running? No, but you’ll get the most out of it if you’re either running one or about to launch.
• Will this work for non-Count agents? Yes. The monitoring framework is tool-agnostic. We’ll demo with Count, but the principles apply universally.
• What’s the difference between this and Part 1? Part 1 is ‘how do I build one.’ Part 2 is ‘how do I keep it good once it’s live.’ Different problems, different stages.