MalloyBench: how well do language models know Malloy by heart?

How much Malloy does a language model know by heart? With the compiler, Claude Opus 5.5 answers all 118 of MalloyBench's tasks. Here each task gets one answer, with no compiler to check against. We ran eleven models at 35 reasoning settings, and half of the benchmark is now public on GitHub.

Results

Claude Opus 5.5 knows Malloy best: at xhigh it passes 111 of 118, and 43 of the 45 hard reports. Below it, value depends on the tier: on the core, Gemini 4 Argon at low passes 82% for 3 cents, and on the frontier GPT-6.1 Sol at xhigh passes 72% for 9 cents, close to Opus 5.5 at high (76%) for less than half the price.

The cheaper models do not get far: none passes a third of the tasks, and on the 74 hard and frontier tasks all of their settings but one pass two or fewer. More reasoning helps the strong models, though not evenly: Opus 5.5 passes 63% at low and 94% at xhigh. Most failed answers never compile, and they repeat the same few mistakes.

What a task looks like

Each task is a question about a dataset, asked with its Malloy model and doc strings, the schema of every source and notes from the data team: about 4,000 tokens. One of the shortest:

run: line_items -> {
  where: ship_date = @1995
  group_by: ship_month is ship_date.month
  aggregate: revenue
  calculate: change is revenue - lag(revenue)
  order_by: ship_month
}

How it works

The six datasets form two halves, each with one dataset familiar from training data, one real but unusual, and one synthetic that no model can have seen. The public half has a TPC-H-derived wholesale supplier, three days of rated Lichess games, and Skerry Line, a fictional ferry operator; the private half a TPC-DS-derived retailer, USDA FoodData Central, and a fictional honey cooperative.

Core tasks (44) cover the language, from joins and nesting to window functions and repairing queries. Hard tasks (45) are reports with eight to ten decisions each. Frontier tasks (29) are easy in SQL and hard in Malloy: funnels, a top N with a row for the rest, gaps in a calendar. Fifteen ask for a Malloy model instead of a query, which hidden queries then test.

An answer passes if it compiles in Malloy's restricted mode, which reaches the data only through the Malloy model, and returns the reference's result on the full data and on two samples of it, so a number typed in from elsewhere fails.

Explore the public tasks

The private half

Public tasks end up in training data, so half of MalloyBench stays private: we run it on released models and publish only the score. The two halves rank the 35 settings alike (Kendall tau 0.88), the public half about 4 points higher, so a model whose public score rises while its private score does not has been tuned on the public half. To have a model scored, open an issue.

Caveats

• One answer per task and setting. In an earlier release, a second run of two settings moved their scores by under a point, but 14 to 25 of their tasks changed outcome.
• One engine: DuckDB, with Malloy 0.0.435.
• October 2026 list prices, with nothing cached. Gemini 4 Argon's are introductory and double on 1 December.

Run it

The harness and the public half are at github.com/Snowboard-Software/malloybench: the code under the MIT license, the tasks and Malloy models under CC BY 4.0. One pass of all 35 settings over the 118 tasks costs about $240 at list price, and building MalloyBench, every test along the way included, cost about $1,500.

Thanks to Lloyd Tabb for reading a draft of this.