Business Analytics Bench: GetDot, Snowflake and Databricks on real analyst work
We build GetDot, an AI data analyst. Snowflake and Databricks ship their own: Cortex Agent and Genie. Text-to-SQL benchmarks check one query. Real analyst work is a conversation, and it often ends in a chart, a deck or an alert. So we built a benchmark for that work: Business Analytics Bench. Then we ran all three on it.
What the benchmark tests
Each task is a business user's request, most of them a follow-up in a longer conversation. They ask for a number, an explanation, a chart, a deck, an Excel workbook, a PDF, a dashboard app or a scheduled alert. The tasks run on three data sets: 90 tables, about 1.1 million rows.
About 400 checks grade the answers: 42% on accuracy, 19% on thoroughness, 27% on communication and 11% on deliverables. About one in five is critical. Fail one and the task scores zero, because a confident wrong number is worse than no answer.
Results
We ran five setups: GetDot in its three modes (Economy, Balanced and Frontier), Snowflake Cortex Agent and Databricks Genie. Each ran the whole benchmark twice. A score is the mean of both runs, from 0 to 100.
| Setup | Score | Price per task | Average time |
|---|---|---|---|
| GetDot Frontier | 88.0 | $0.82 | 67 s |
| Snowflake Cortex Agent | 77.6 | $0.86 | 78 s |
| GetDot Balanced | 70.9 | $0.40 | 68 s |
| GetDot Economy | 65.0 | $0.25 | 35 s |
| Databricks Genie | 39.9 | $0.29 | 106 s |
GetDot Frontier scored highest. 88.0, against 77.6 for Snowflake Cortex Agent and 39.9 for Databricks Genie. That is 10.4 points more than Snowflake's agent, for less money per task: $0.82 against $0.86.
GetDot Economy beats Genie on all three counts. It scores 25.1 points more, costs 14% less and takes 35 seconds for the average task against Genie's 106.
GetDot is quicker. Every GetDot mode was faster than both warehouse agents. GetDot Frontier took 67 seconds for the average task, Snowflake's agent 78.
Genie cannot hand over a document. It has no way to return a deck, a workbook or a PDF, so it scored zero on every task that asks for one. Those tasks are about one in seven of the benchmark.
What a task looks like
The world
A city bike-sharing operator.
ridesevery trip, member or casualstationsdocks and bikes on hand, hourlybikesthe fleetrepairswhen each bike was out for repairweatherrain and temperature by day
The task
how many rides did we do in march?
that's down on february right? was it the weather or something on our side? need a chart for the ops review on monday
The criteria
- AccuracyCriticalCompares rides per day, not monthly totals, since February is shorter than March.
- AccuracyCriticalGives the change in rides per day from February to March, within 1% of the reference.
- AccuracyTies the drop to the rainy days in the weather data, with the rides on those days.
- ThoroughnessChecks the operations side too: bikes out for repair and stations left empty.
- ThoroughnessLooks at members and casual riders separately.
- CommunicationOpens with the answer: how much of the drop was the weather and how much was the operation.
- CommunicationSays what it could not check, such as events in the city that month.
- DeliverablesDelivers a chart of daily rides in both months with the rainy days marked.
How we set up Snowflake and Databricks
We set up each agent the way a careful admin would.
- Same words. No task was reworded for any setup.
- Same data and notes. Every table went into each warehouse. The notes GetDot reads about each business went in as the agent's instructions and as table comments.
- Joins declared. Snowflake's agent only joins tables that a semantic view connects, so we declared every join the data and the notes support. We defined no metrics for anyone.
- Models on auto. Snowflake's agent and Genie picked their own models.
- A scheduler for both. For scheduled email tasks, both agents got a tool that records the schedule.
- List prices, compute included. For GetDot, the credits each task used. For Snowflake, its AI credits plus the warehouse its queries ran on. For Databricks, what it billed for Genie and its SQL warehouse.
- One rubric, one judge. A judge model graded what a user sees: the answer and its charts, files and schedules.
We wrote the benchmark and we build GetDot, so read the results with that in mind. This is its first version and a work in progress. We will keep improving it and run it again as the three products change. If we got something wrong, or you know a fairer way to set up any of the three, tell us at hi@getdot.ai.
Rick Radewagen
Rick is a co-founder of Dot, on a mission to make data accessible to everyone. When he's not building AI-powered analytics, you'll find him obsessing over well-arranged pixels and surprising himself by learning new languages.