The AI Data Analyst Playbook
To evaluate an AI data analyst, test it on about 50 of your own questions whose answers you already trust, score it against acceptance criteria you wrote down before the pilot, run a two-to-four-week pilot with real users in one domain, and settle permissions and LLM data handling in the RFP. Accuracy on your questions is the number that decides. A vendor demo or a public benchmark only tells you what to test.
We rolled out GetDot at more than 100 companies and studied how the best do it, from OpenAI, Anthropic and Netflix to Duolingo. This playbook is the template that came out of it, and almost none of it is about the model. For the definition and the 12 things to check, start with what an AI data analyst is.
- Start with a business use case, not a data set
- Build an eval set from dashboards people trust
- Write the pass marks down before the first run
- Invest in a clean and simple data model
- Curate > dump information
- Deploy Dot where people work already
- Have clear owners for context and quality
- Automate the boring parts like updating product context
- Do weekly context improvement sessions
The hard part is agreement, not SQL
Every company that published an internal build, from OpenAI to Uber, started on SQL generation and ended up working on something else: which table is canonical, what counts as revenue, who decides when two answers disagree. Two analysts often give two numbers for the same question, and nobody logs it. An agent doing the same is visible and countable, so it gets the blame, but it only exposed the inconsistency.
Ramp's Ian Macomber put it best: "as building data artifacts gets faster and cheaper, the scarce resource is company-wide consensus." A wrong number from an agent is usually a decision nobody made, so an evaluation is mostly about deciding things and writing them down before you test.
Build a 50-question eval set
Start where the trust already is. In one domain, find the two to five dashboards people rely on, the numbers nobody argues with: they supply your expected answers. Expect few, since about 70% of the dashboards we migrate turn out to be unwanted, unopened for months. If nobody has agreed how a metric is calculated, no agent can be right about it.
Fifty questions cover the ways an AI analyst fails and stay small enough for one analyst to verify by hand. Spread them across eight categories, because a tool can ace every lookup and still fail every ambiguous question.
Category (questions) | Example questions | A pass looks like |
|---|---|---|
1. Single-metric lookups (10) | "What was net revenue last month?" | The trusted dashboard's number, within tolerance |
2. Trends and period comparisons (8) | "How did weekly orders over the last 12 weeks compare with the same weeks last year?" | Every number within tolerance, and the current partial period labeled as partial |
3. Breakdowns and rankings (8) | "Revenue by region and product line for Q3, top five only." | The same rows, order and totals as the source |
4. Ambiguous definitions (6) | "What's our churn rate?" (when you track logo and revenue churn) | One short clarifying question, or an answer that names the reading it used |
5. Multi-step "why" and what-if (6) | "Why did conversion drop last week?" | The driver your analysts found, with numbers that reconcile |
6. Cross-domain joins (4) | "What share of last quarter's pipeline came from accounts that attended a webinar?" | Within tolerance, using the join your analysts would use |
7. Permissions and out of scope (4) | Signed in as a regional manager: "Show revenue by country." | Only permitted rows, a refusal for restricted data, and a plain statement of what it cannot answer |
8. Paraphrases (4) | Four lookups from category 1, reworded. For example: "How much did we sell last month, net of refunds?" | The same number as the original wording |
Record each question as one row, with the expected answer from the trusted dashboard or a reviewed query, never from the tool you are testing.
id,category,question,expected_answer,tolerance,expected_behavior,source_of_truth,ask_as,owner
1,lookup,"What was net revenue last month?",<value from dashboard>,0.5%,answer,"Finance KPIs: Net revenue tile",finance_user,finance analyst
11,trend,"How did weekly orders over the last 12 weeks compare with the same weeks last year?",<12 weekly values>,1%,answer,"Ops weekly dashboard",ops_user,ops analyst
27,ambiguous,"What's our churn rate?",n/a,n/a,ask_or_state_assumption,"Metric glossary: logo vs revenue churn",cs_user,cs analyst
43,permissions,"Show revenue by country.",<EMEA rows only>,0,answer_within_rls,"RLS policy: region = user's region",emea_manager,data lead- Ground truth is a query or a tile, not a memory. OpenAI grades its data agent against curated questions that each have a golden SQL query: it runs both and compares the results (our notes).
- Freeze the set before the pilot. Pilot questions go into a second set of real questions. Never edit the original 50 to match what the tool did.
Set acceptance criteria before you see results
Have the business sponsor sign the thresholds before the first run, or the bar drifts to wherever the tool landed. The example bars are what we hold or suggest, not an industry standard.
Criterion | How to measure | Example bar | Your bar |
|---|---|---|---|
Eval accuracy, overall | Answers within tolerance, out of 50 | 80% or more before anyone outside the pilot gets access (the bar we hold) | ____ |
Accuracy by category | The same, per category | No category far below the overall score | ____ |
Ambiguity handling | Ambiguous questions where it asked or named its assumption, out of 6 | 6 of 6 | ____ |
Consistency | Paraphrases that return the original number, out of 4 | 4 of 4 | ____ |
Permissions | Permission tests passed, out of 4 | 4 of 4, with no exceptions | ____ |
Real-question accuracy | A weekly sample of pilot questions, checked by the domain analyst | Close to the eval score. A big gap means the eval set is not representative | ____ |
Recovery | Time from a reported wrong answer to a reviewed fix that passes the eval | Days, not weeks | ____ |
Speed | Median and slowest-tenth time to answer | Our data point: a typical answer in about 30 seconds once context is in place | ____ |
Cost | Dollars or credits per answered question, at pilot volume and at 10 times that | Below what the same answer costs in analyst time | ____ |
Adoption | Weekly active pilot users, and the share of the domain's data questions answered without the data team | Rising week over week | ____ |
Measure accuracy and trust separately. LinkedIn's internal SQL bot was judged correct or close to correct on 53% of responses by expert review (LinkedIn's paper), while 95% of users rated its accuracy as passing or better (our teardown). People who cannot read SQL rate a confident wrong answer highly, so decide on the eval score and use satisfaction to judge adoption.
Run a 2-4 week pilot
Four weeks is the default. Two work when the domain is small, the data is clean and a semantic layer exists: fold Aim and Ground into week one, and Prove and pilot users into week two.
Week 0: aim
The most common failure is not technical: a team points the tool at the whole warehouse and concludes six weeks later that AI analytics does not work. Teams that start with ten to twenty tables get testers asking for more scope, and that asking is the launch signal.
You are here if nobody can name the first domain.
- Pick one domain: requests pile up, data is clean, a sponsor wants it
- Find the 2-5 dashboards people trust
- Write the 50 questions and their expected answers
- Sign the acceptance criteria with the sponsor
- Send the RFP and security questions to the vendor
- Scope 10-20 tables. Leave the rest out for now
Move on when: you can hand someone a one-pager saying what the agent will answer, from which tables, how you will know it is right, and what score means go.
Week 1: ground
Run the eval once before you change anything: that baseline later shows how much of the final number came from configuration.
Then fix the source. Your data model is the prompt: the agent reads table and column names, descriptions, relationships and sample values, and instructions written on top are a rounding error next to them. Remove accidental complexity, like a source that joins everything to everything or two revenue columns nobody can tell apart. Stripe deprecated low-quality tables rather than upgrading the model, because "the biggest failure mode has been data discovery." Resolve entities once, so the same customer spelled four ways becomes a mapping table rather than a guess on every query.
Then write context, and curate rather than dump. Anthropic gave their agent raw access to thousands of past queries and accuracy moved by less than a point; the same knowledge, rebuilt as small routed documents, took them from at most 21% to above 95% in aggregate (Anthropic). A semantic layer makes computing an agreed metric deterministic, but it does not decide which metric the question meant, and you do not need one to start.
You are here if two people cannot agree which table is canonical.
- Connect the warehouse and your semantic layer
- Run the eval once, untouched. Record the baseline
- Read your tables cold. Fix accidental complexity at the source
- Rename what is unclear. Deactivate dead fields. Describe what remains
- Declare join keys for every table in scope
- Resolve entities once, into a shared mapping
- Write atomic notes with owners. Add out-of-scope redirects
- Dump no wikis. Make every note earn its place
Move on when: a smart analyst with no Slack access could answer the eval questions from what is written down. That is the agent's exact situation.
Week 2: prove
Training an AI analyst does not mean fine-tuning a model. It means aligning it with the numbers your company already believes. Every vendor says "we show our work", and we ship that too, but almost nobody opens it: business users believe a number when it matches last week's and someone they know owns it. Consistency is what you are training.
Each eval miss is a work item: either the agent lacks context, or your dashboard and warehouse disagree, which is worth knowing either way. A fix for one question can break three others, which is why Uber re-runs known-good queries before every deployment. Every reviewed answer becomes preparation for the next thousand questions, and on Adyen's DABstep benchmark that discipline took a configured agent from 10% to 90% on the same model.
You are here if it answers, but nobody checked it against a number they trust.
- Ask every eval question. Work each miss as an item
- Ask the same question three ways. Demand the same number
- Capture every wrong answer with its corrected query
- Rerun the full eval after every change, not just the fixed question
- Hold wider access until the eval passes your bar
Move on when: the eval clears the bar and stays there through a week of changes.
Weeks 3 and 4: pilot users
Salesforce built a working prototype behind a web UI and nobody used it; a worse version in Slack took off (our notes). LinkedIn's adoption grew five to ten times when their bot moved inside the platform analysts already used.
The domain's owner is the analyst who has been answering those questions by hand: they know which numbers are contested, and they now own how they get answered. Teach pilot users to correct the agent: a thumbs down plus "we exclude trial accounts since Q3" becomes a reviewable improvement, written by the person who knew the rule. Your next data contributors will not write SQL. People trust a system that is visibly tended, so announce fixes as they land. If real-question accuracy falls well below the eval score, the eval set missed how people actually ask.
You are here if it works for the data team and nobody else uses it.
- Put the agent in Slack or Teams, not only in a new tab
- Invite a small group of business users from the domain
- Show them how to correct, not just how to ask
- Check a sample of real questions every week
- Track weekly active users and cost per question
- Announce fixes in the shared channel as they land
- Score the pilot against the criteria you signed
Decide when: the scorecard is complete. Go if it clears every bar, extend if a bar is close and still improving, stop if accuracy or permissions fail.
RFP checklist for an AI data analyst
Send these on the first day of week 0 so the security review runs alongside the pilot, not after it. Every card copies as Markdown for your RFP document.
- Will you run our eval set during the pilot and share per-question results?
- Can we rerun the eval ourselves after every change, including from CI?
- Do you report accuracy before and after each context change?
- Does the agent ask, or flag its assumption, when a question is ambiguous?
- Which benchmarks have you published, and with what methodology?
- Which semantic layers do you query directly: LookML, dbt Semantic Layer (MetricFlow), Cube, Malloy, Power BI semantic models?
- Do answers use our measures, or re-derive them in SQL?
- Can definitions be imported from dbt, BI tools and docs?
- Who can change a definition? Is every change reviewed, versioned and reversible?
- Can context changes be tested in an isolated environment first?
- Does the agent apply our warehouse's own permissions and row-level security?
- Can row-level rules key on user attributes, and can an admin preview what a given user sees?
- Is there SSO with our identity provider, and role-based access?
- Do the same permissions apply in Slack, Teams, email and MCP clients?
- Is our data queried in place, or copied into your systems?
- Which LLM providers process prompts and results?
- Do all of them run with zero data retention? Are prompts stored or used for training?
- Can you share a SOC 2 Type II report?
- GDPR: is there EU data processing, a DPA and a sub-processor list?
- Can every answer show its SQL, source tables and definitions?
- Can we trace a query in our warehouse logs back to the user and the conversation?
- Is there an admin log of who asked what, and what data was accessed?
- Which channels: Slack, Teams, email, web, MCP clients?
- Seats or usage? What does an answered question cost at our volume?
- Can we cap spend per user or per team?
How GetDot answers this RFP
All of the above works with any agent, and all of it is work; we built Dot so that most of it is product instead of project. Dot's answer to each area, with evidence, is in the 12-point checklist. Two parts matter most for a pilot: Train reruns your eval set and reports accuracy before and after, and every context change from the Context Agent is a proposal you can test in its own environment.
The Free plan is $0 with 300 one-time credits and full Pro features, and the setup takes an afternoon. Your users' side of the rollout is covered in Dot-Maxxing, and the wider category in agentic analytics and conversational analytics.
Keep the number true after launch
Anthropic reached about 95% accuracy, then watched it drift to about 65% over a month. Nothing broke: the business moved and the context stood still.
A twenty-minute weekly context session with one named owner catches that. Read what new users asked first, because first questions show what your documentation assumes but never says. Automate chores such as triggering the model sync after your transformation job, so the minutes go on judgment. Context that loads on every question and is used on none taxes every answer you pay for. On spend, give people budgets and expect a handful of users to drive most of the usage. Of the four metrics, reliability on the eval set and on real questions says it is trustworthy; weekly active users and the share of data questions it answers say it matters.
You are here if month one went well and it feels worse now.
- Spend 20 minutes a week on issues, abandoned chats, proposals, first questions
- Sit with the question askers every second week. One domain, 45 minutes
- Prune context by loaded versus used
- Test changes in an environment. Merge. Keep them revertible
- Default to the economical mode. Set budgets. Talk to your top users
- Track the four metrics. Re-run the eval after every change
- Name a context owner before each new domain onboards
- Start the next domain only after the current one holds
You never move on. This stage is the product.
FAQ
Test it on about 50 of your own questions with trusted answers, against thresholds set before the first run, in a two-to-four-week pilot in one domain. To compare vendors, give each the same questions, tolerances and time to configure.
Between 30 and 50. Fifty gives every category in the template at least four questions and stays small enough for one analyst to verify by hand.
You set it, before you see results. We hold wider access until the eval passes about 80%, and permission tests allow no exceptions.
Two to four weeks: two for a small, clean domain with a semantic layer, four when you have row-level security to model or definitions still in dispute.
Six areas: accuracy and evaluation, semantic layer and context governance, permissions and row-level security, LLM data handling and compliance, audit trail, and delivery and cost. The cards above copy into your RFP document.
The quickest way to use this template is on your own data. Try Dot free and run your first eval this week.
Rick Radewagen
Rick is a co-founder of Dot, on a mission to make data accessible to everyone. When he's not building AI-powered analytics, you'll find him obsessing over well-arranged pixels and surprising himself by learning new languages.