The AI Data Analyst Playbook

We rolled out Dot at more than 100 companies. We also studied how the best do it, from OpenAI, Anthropic and Netflix to Duolingo. This playbook is what works.

Almost none of it turned out to be about the model. Here is the whole thing in eight lines.

The hard part is agreement, not SQL

Every company that published an internal build, from OpenAI to Uber, started by working on SQL generation. All of them ended up working on something else: which table is canonical, what counts as revenue, who decides when two answers disagree.

You can watch the real problem in your own company today. Ask two analysts the same question and you will often get two numbers. Nobody logs that, so nobody counts it. An agent doing the same thing is fast, visible and countable, so everyone blames the agent. It did not create the inconsistency. It exposed it.

Ramp's Ian Macomber put it best: "as building data artifacts gets faster and cheaper, the scarce resource is company-wide consensus." A wrong number from an agent is usually a decision nobody made, not a model that failed.

So this playbook is mostly about deciding things, writing them down, and keeping them true while the business moves.

Start where the trust already is

The most common failure is not technical. A team buys a tool, points it at the whole warehouse, and concludes six weeks later that AI analytics does not work. Nothing was wrong except the aim.

Start with a business use case, not a data set. Pick one domain where requests already pile up, the data is reasonably clean, and someone is asking for it. Two of those three is enough.

Then find the trust that already exists there: the two to five dashboards people rely on, the numbers nobody argues with. Expect that set to be small. About 70% of the dashboards we migrate turn out to be unwanted. Nobody had opened them in months. Write down the questions those dashboards answer and the answers they give. That list becomes your eval set, and it is the most important artifact of the rollout. There is no accuracy in a vacuum. If nobody has agreed how a metric is calculated, no agent can be right about it.

Teams that connect the whole warehouse stall. Teams that start with ten to twenty tables get testers asking for more scope. That asking is the launch signal.

Fix the source before you write anything down

What does the agent read when it answers? Table names, column names, descriptions, relationships, sample values. Your data model is the prompt, and the instructions you write on top are a rounding error next to it.

So invest in a clean and simple data model before you write a word of documentation. Read your tables the way the agent will: cold, literal, no folklore. Every business has real complexity it has to model. Most warehouses also carry accidental complexity, like a source that joins everything to everything, or two revenue columns nobody can tell apart. Fix that at the source, deleting where you can. Stripe deprecated low-quality tables rather than upgrading the model, because "the biggest failure mode has been data discovery." Vercel put it in one line: on a messy data layer, a better agent just gets you "faster bad queries." Declare how tables join, and resolve entities once, so the same customer spelled four ways becomes a mapping table rather than a guess on every query.

Then write context, and curate rather than dump. Anthropic gave their agent raw access to thousands of past queries and accuracy moved less than a point. The same knowledge, rebuilt as small routed documents, took them from 21% to over 95%. Same words, different structure. One topic per note, an owner on every folder, and nothing pasted in wholesale.

If you already run a semantic layer, connect it. It makes computing an agreed metric deterministic, which is worth having. It does not decide which metric the question meant. If you do not have one, you do not need one to start.

Train against numbers people already trust

Training an AI analyst does not mean fine-tuning a model. It means aligning it with the numbers your company already believes.

Do not expect transparency to do this for you. Every vendor answers the trust question with "we show our work." We ship that too, and almost nobody opens it. Business users believe a number when it matches what it said last week and when someone they know owns it. Consistency is what you are training here.

Ask it the questions your trusted dashboards answer, and compare. Every disagreement is a work item: either the agent is missing context, or your dashboard and your warehouse disagree, which is worth knowing either way. Ask the same question three ways and demand the same number.

Keep those questions and their verified answers as a permanent eval set. OpenAI grades generated SQL against golden queries by running both and comparing results. Uber re-runs known-good queries before every deployment. Without an eval set you cannot tell when a change made things worse.

When an answer is wrong, capture the correct query beside it. A wrong answer plus its ground truth teaches more than any description of the failure, and every reviewed answer becomes preparation for the next thousand questions. Ship when the eval passes your bar, not when the demo feels good. On Adyen's DABstep benchmark, that discipline took a configured agent from 10% to 90%.

Launch where people already work

Salesforce built a working prototype behind a web UI and nobody used it. They shipped a worse version into Slack and adoption took off. LinkedIn's adoption grew five to ten times when their bot moved inside the platform analysts already used. Deploy where people work already.

Then give every domain a clear owner, and you already know who it is. It is the analyst who has been answering those questions by hand. They know which numbers are contested and which request comes back every month. They are not losing that work. They now own how it gets answered.

Teach users one behavior above all others: correct the agent when it is wrong. A thumbs down plus "we exclude trial accounts since Q3" becomes a reviewable improvement, written by the person who knew the rule. Your next data contributors will not write SQL. Announce the fixes in the shared channel as they land, because people trust a system that is visibly tended.

Then expand one domain at a time, each through the same gate.

The chapter everyone skips

Anthropic reached about 95% accuracy, then one month of ordinary schema and definition changes dropped them to 65%. Nothing broke. The business moved and the context stood still.

What keeps a system trusted is small: a weekly context improvement session. Twenty minutes, one named owner. Work the conversations flagged as problems and the ones users abandoned, approve or reject the pending improvements, and skim what new users asked first, because first questions show what your documentation assumes but never says.

Automate the boring parts. Trigger the model sync after your transformation job so the agent is never a day behind, and refresh the context that goes stale on its own schedule. The twenty minutes should go on judgment, not chores. Prune with evidence too: context that loads on every question and gets used on none is taxing every answer you pay for. Make changes in an isolated environment and keep them revertible.

Govern the spend deliberately. Set an economical default, give people budgets, and expect a power law, because a handful of users will drive most of the usage. More usage costing more is fine. It means people are getting answers.

Then measure four things: reliability on your eval set, reliability on real questions, weekly active users, and the share of the company's data questions the agent now answers. The first two say it is trustworthy. The last two say it matters.

Where Dot fits

Everything above works with any agent, and all of it is work. We built Dot so that most of it is product instead of project: training against trusted dashboards, evals, per-answer lineage, correction-to-proposal review, version-controlled environments, spend controls. The setup takes an afternoon. Your users' side of the rollout is covered in Dot-Maxxing.

If this excites you, we'd love to hear from you. Get in touch.