MalloyBench: language models write Malloy well with its compiler

We build Dot, an AI data analyst, and we like Malloy enough to build it in. Malloy, from the team that built Looker, is a semantic modeling and query language: a model says once how tables join and what each metric means, so a query can ask for revenue by region without restating either. And Malloy is strongly typed: its compiler checks a query against the model before anything runs, and says what is wrong and where. For an AI agent, that is a feedback loop of seconds.

MalloyBench measures which language models write Malloy well, on 118 data tasks that are hard on purpose so that it does not saturate. Here a model works as an agent in Dot does, with the compiler and the database at hand. We ran eight models at 18 reasoning settings, all on standard tiers so that their times compare.

Results

Claude Opus 5.5 at high effort answers all 118, in a median of 33 seconds for 20 cents. GPT-6.1 Sol comes close for a quarter of the price: 114 at medium, in 26 seconds. Gemini 4 Argon is nearly as accurate but takes two minutes, and the cheapest settings, Claude Haiku 5.5 and GPT-6 Luna at high, pass about half for 1 to 2 cents.

The strong models need few steps: Opus 5.5 three an answer and Sol under five, typically one run to look at the rows. The models that pass about half take eight to fourteen, and a minute or more.

What the compiler adds

Every setting passes more with the compiler than from memory alone: the median setting gains 35 points, and GPT-6.1 Sol the most, from 42% to 97%.

Asked for each month's revenue in 1995 and its change from the month before, Claude Sonnet 5.5 at low effort took three tries and 6 seconds:

The compiler's first reply names the fix; the second is a compiler bug in ~ @1995 (#3162), which the model writes around. What still fails is rarely syntax: one failed answer in a hundred does not compile. Most return the wrong result, which no compiler can catch, and most of the rest run out of steps. Without a compiler, models repeat the same few mistakes, and Malloy could absorb several of them.

How it works

The tasks are MalloyBench's 118, on six datasets in three tiers, and an answer passes when it returns the reference's result on the full data and on two samples of it. A model gets the Malloy model and the schema of every source, about 4,000 tokens, and three tools: compile a query to see its SQL or its errors, run it to see the row count and up to 20 rows, or submit it. A task ends at submission or after 12 tool calls.

Cost is the October 2026 list price, each step reading the previous one's prompt from the cache as an agent's would; Gemini 4 Argon's price is introductory and doubles on 1 December. Time is the model's own, summed over its steps, median over the tasks. Each setting answered each task once.

The harness and the public half of the tasks are at github.com/Snowboard-Software/malloybench, where --tracks agentic runs this setup.

Thanks to Lloyd Tabb for reading a draft of this.