MalloyBench: what Malloy could improve for language models
Language models write Malloy well when its compiler can tell them what is wrong. From memory alone, eleven models at 35 reasoning settings failed 2,826 answers to MalloyBench's 118 tasks, mostly in the same few ways: 84% of those answers never compile, and 11% run but return the wrong result.
The compile errors are rarely deep: SQL habits, a few Malloy rules, and older versions of the language.
Five mistakes make up nearly two fifths of the failures:
- A reserved word as a name:
month,yearordayhas to be quoted, and every model forgets, Opus 5.5 included. - A function Malloy lacks, most often
median(), called 149 times. - Old syntax:
project:,from(), or a source withoutextend. count(distinct x), an error since Malloy 4.0, wherecount(x)already counts distinct values.- SQL window functions:
sum(x) over (partition by y), which Malloy writes incalculate:.
What a language reference fixes
malloyyo, the Malloy team's server for AI clients, ships a 37,000-character language reference. We put it in the prompt of the 19 cheaper settings and ran them again.
It helps on the easy tasks, a median 20 points on the core against 2 on the hard and 3 on the frontier; Claude Haiku 5.5 at high doubles from 20% to 41%. It removes the mistakes it spells out: count(distinct x) falls from 121 failed answers to 2, and old syntax from 92 to none. It says nothing about reserved words or a median, and neither falls; adding both would be easy. What a reference cannot teach is how to plan a report with ten decisions.
What the compiler and the language could change
For several of these mistakes the compiler already names the fix, such as quoting month. Where a message points away from the fix, models go in circles: we reported four such messages in #3164, among them project:, which reports a missing colon.
Malloy could also absorb a few of these mistakes without changing what a correct query means: let a reserved word name a field wherever it cannot be read as anything else, accept count(distinct x) with a warning, and add a median. Alone they would rescue few answers, because an answer with one mistake usually has others: quoting the reserved word passes 24 of 344, and writing count(x) 28 of 150.
Compiler bugs we found
Writing the 118 reference answers and checking each against plain SQL turned up nine compiler bugs. We reported the six new ones; four of them return a wrong number with no error, which an agent cannot catch.
| What happens | Upstream |
|---|---|
!~ with a filter expression matches instead of excluding | #3159 |
? with a regular expression compares text | #3160 |
exclude() of an outer dimension in a nest returns the largest group's value | #3161 |
having: on a dimension empties all() | #3163 |
~ with a time or number range crashes the compiler | #3162 |
| Subtracting two dates reports an error about a logger | #3164 |
count() through one join_many counts parents with no match | open since 2024, #1762 |
| A join condition that uses a dimension defined through a join breaks the SQL | open, #2486 |
A limit: refinement drops the view's order_by: | fixed in 0.0.436, #3136 |
Each has a self-contained repro on DuckDB's TPC-H data in docs/malloy-issues.md.
Thanks to Lloyd Tabb for reading a draft of this.
Rick Radewagen
Rick is a co-founder of Dot, on a mission to make data accessible to everyone. When he's not building AI-powered analytics, you'll find him obsessing over well-arranged pixels and surprising himself by learning new languages.