I recently rolled out an analytics agent that allows internal users to ask questions about the business in plain English and get back results based on the data stored in the data warehouse. (View Highlight)
When I initially connected the agent to the data to showcase it to the data team it did not do a great job at answering basic questions, but we did not have a baseline about how accurate answers were, we decided to build a way to measure it and make incremental fixes to improve accuracy. (View Highlight)
TLDR; our baseline started at 60% and improved to 98.6% after a few days, this post explains step by step how to get there (View Highlight)
Before putting the analytics agent in front of real internal users, I wanted to get a straight answer to a simple question: can people trust what the agent is returning? (View Highlight)
An agent that is wrong a high percentage of the times is worse than no agent at all, because wrong answers are hard to spot unless the person has lots of domain knowledge and the right expected ballpark for the question asked, given that the majority of people using the agent are not technical and are not familiar with the underlying data model giving them those details won’t help. (View Highlight)
An agentic eval is a repeatable check that runs your agent on a set of inputs and scores the output against what a good answer looks like. Think of it as a test suite for a system whose behavior you cannot read from code and that is not deterministic (View Highlight)
Our core company KPIs: This set was taking on what we already reported and common follow up questions from past business review meetings on changes on those metrics (View Highlight)
Recent ad-hoc questions from our support channels: we kept the messy real world phrasing, with ambiguous terms and implicit context, that no one writes in a spec (View Highlight)
Recent reported issues with the data model: when we got feedback or asked about which table to use to answer a question because the data model was not clear (View Highlight)
• LLM-based agents are non-deterministic, ask the same question twice and you may get different ansers
• Analytics agents are generally multi-turn, you ask an initial question and you follow up with more to dig deeper and get to the root cause
• Internal user may use business jargon or language that may not be present in the data model
• Edge cases: Data might not be there to answer the question or the data may be affected by incidents in the source
• Questions may not be well scoped, ambiguos or clear in the first go
• Not only accuracy is important, but how it got there as well (cost, time to answer, steps and tool calls used, consistency etc)
• Running the full eval set with every change is too expensive (View Highlight)
Give evals some level of determinism (for example, fix the time frame for a time in the past) so that it is easier to compare with real figures (View Highlight)
Include single turn and multi-turn evals so that the agent is able to exercise asking follow up questions when ambiguos answers are asked and flag context management or caching issues (View Highlight)
Include cases that asked about data that was missing and where the data exists but is unclear, this help evaluate that edge cases get exercised and verified (View Highlight)
Get an LLM to grade answers (LLM as judge), the judge model is different from the model evaluated and should return yes/no answer (View Highlight)
• Trajectory was left for debugging purposes but not evaluation, getting an accurate answer is more important than how the agent got there (with a couple of small exceptions, for example, filling tickets when data is missing is a requirement rather than a nice to have, I want to ensure it happens)
• Run the full set of evals daily and a sample on CI
• Each eval should be run multiple times to measure consistency of answers (View Highlight)
Restrict the agent to data that is trustworthy: Only the datasets that are used day to day to answer questions or plot dashboards, these are battled tested and have a lot of visibility, everything else was removed. This makes it less likely for the agent to pick a testing dataset or one that was ad-hoc done (View Highlight)
Give profiling and data transformation tools: equip your data agent with this tools to flag untrusted datasets and understand how data is flowing from raw data to metric (View Highlight)
Create a business glossary with metric definition and common business terms, acronyms and synonyms and give the agent access to it. This removes ambiguity and helps the agent choose the right data source right away. (View Highlight)
Writing better descriptions for tables and columns as well as following good naming conventions for those, a column called amt_2 with no description makes the agent guess and more likely to pick the wrong ones for answering questions. Think about including, grain of the table, what it can be used for (and what it can’t be used for), how to leverage the dataset to query it including real world example queries. (View Highlight)
Agentic evals high level article from Yesha Shah, this was a great inspiration for this article (Link here) (View Highlight)
• Measure before you ship. Without a score, you cannot tell whether a change helped.
• Treat evals as non-deterministic from the start. Run cases multiple times and look at rates, not single results.
• Track cost and latency, not only accuracy. A more accurate agent that is slow and expensive could be considered a regression
• Fix the context, not just the prompt. Trusted data, clear descriptions, and a shared glossary did more for us than any clever instruction.
• Turn every failure into a test. That is how a suite becomes a safety net.
• Test edge cases. A good agent knows when to hand a question to the data team instead of guessing.
• Keep some questions out of your tuning. Otherwise a better score may only mean the agent learned the test. (View Highlight)
This helped move accuracy from 60% to 98.6% by getting better at measuring and then fixing what the measurements showed. But this is not a one-off project. The data model keeps evolving, and new metrics are added, giving the business more visibility into its data is an ongoing process as the product evolves. Every one of those changes means adapting eval cases: new metrics need new cases, changed definitions need updated expectations, and requires the team to keep checking that the answers stay accurate as the data moves underneath the agent. (View Highlight)
