Burning a Billion Tokens to Try to Beat SQL
Burning a Billion Tokens to Try to Beat SQL
In a companion post we argue agents have created a golden age of infra UX development - you can force agents to run repeated reps in a way that would be inhumane for your testers; you control the context they see - mandatory doc reading! - and these evaluations can even be embedded in a standard release process if you're tolerant of a small CI token bill.
This makes UX an optimization problem more akin to performance or reliability, where we can repeat cycles of measuring, improving, and measuring again - though again, with a token cost. Optimistically, inference prices continue to collapse - especially at the sub-frontier level you want to measure - and this is an easy cost to pay.
That post didn't spring from a vacuum - we learned its lessons by confronting our own language hubris as we tried to benchmark Trilogy to prove how agent-friendly it was.
Trilogy was built for humans, not agents - but we sure expected it to be good for agents, better than raw SQL. Early agents seemed to validate this, but were generally too inconsistent to be reliable. We could see the promise, but those were the days of 'you can't trust an agent to write SQL' (a take that has aged shockingly poorly).
In late 2025, agent capabilities achieved lift-off; in 2026, we attempted to empirically validate that we were, in fact, 'better than SQL'.
Tips
I cannot emphasize how good leading-edge models are at SQL today. With some limited ability to explore and a properly specified question, they will get it right. Their Achilles' heel is what it always was in a messy enterprise data environment - nothing is actually well specified, the warehouse is a burning garbage heap where most things are out of date, and someone decided it would be cute to name all their staging tables after varieties of European birds.
Obviously, things didn't go as planned. A billion tokens - or approximately $40 - later, here's the story of how it went:
First, some context
Trilogy is SQL without the tables: you write SQL against a metadata layer that late-binds to physical tables to fulfill your query. This has a lot of nice properties, many tailor-made for agents:
- Rich context by default, with tools for progressive disclosure.
- No fan-out and chasm traps; fewer sources of error
- Automatic aggregate resolution; performance for free
- Strong typing and linting for fast feedback.
- All the power and expressiveness of SQL; not constrained like other semantic layers.
Day 0: Disappointment
We stood up a scrappy eval harness and pointed it at a model we knew well - TPC-DS. We'd run 4 legs:
- Raw SQL connection
- Raw SQL + a schema dump
- Auto-ingested Trilogy model
- Hand-crafted Trilogy model
The eval harness was the same agent with a slightly different toolset for both legs; SQL got an SQL tool and might get a schema dump up front; Trilogy got the Trilogy CLI, with a few things locked down or redacted.
For questions, we roughly backported a plausible "business scenario" from looking at DuckDB's TPC-DS extension reference SQL: "What question could someone have asked that this would answer?"
We'd been generating Trilogy for a while and had an agent we thought would be in the sweet spot for cost - DeepSeek Flash.
We kicked off the run, grabbed a coffee, came back.
Headline:
SQL legs: ~60% pass rate! Not bad. Trilogy: ~30% pass rate, and a lot more tokens.
Not the resounding win we expected. Pretty awful, in fact.
Step 1: Reality Check
If you write a language, you have a lot of implicit context. If you are instead trying it for the first time, you will hit every footgun. Our happy path testing was not what agents did - they blundered into every sharp corner of the language and had no slack channel to go ask for help in. Anything not explicitly spelled out would be learned the hard way or misused. There were a lot of weird stack traces and errors.
We buckled down, expanded fuzz testing, worked on linters, agentic guidance, etc.
Each run would claw back a few percent here, a few percent there - we were getting up to ~50% pass rate, with reduced tokens.
Final tally: +20% pass rate on basic hygiene. Lesson learned, and a pretty happy outcome - it is much less painful to have DeepSeek indirectly rip your language apart by being terribly confused by syntax than it is to make your friends do it.
The score: Trilogy ~50%, SQL ~60%.
Step 2: TPC-DS, What the Hell
At this point, as we started looking at failing Trilogy cases, we started to realize that TPC-DS is designed as an engine stress test and has some truly weird questions. Worse, many of them returned no results at all on the scale factors we used for testing.
We buckled down, made our questions more precise, gave overrides for ambiguous language, rewrote some prompts, fixed some buggy queries where agents would reject the right answer as too odd. This at times somewhat required violating our 'someone in a business could ask this question naturally' guideline, but that seemed reasonable in the spirit of testing agentic generation capabilities - maybe agents would specify that they'd like nulls last, please.
A more interesting learning was that we had straight screwed up with our human-crafted semantic model - we didn't really understand the dataset fully, and our auto-ingested one had more accurate context in some places. Some tweaks brought humans back in the lead, but this was an early signal that automatic model ingest would be able to get to higher precision.
This clawed back about another 20%. You can't trust a bad eval. Another lesson learned! We're really collecting those scout badges.
In terms of Trilogy vs SQL, this was treading water - making questions better lifted both legs.
Scoreboard: Trilogy ~70%, SQL ~80%.
Step 3: Language Features
The remaining gap between Trilogy and SQL stemmed from more exotic features stressed by TPC-DS - rollups, unions, etc. Many of these could be expressed in more convoluted ways without those constructs, but doing so was unnatural and agents would rarely reach for them.
So we targeted these one by one - rethinking rollups; enabling anonymous concepts in select outputs; adding in unions as a generic table-valued function implementation.
One of the biggest language changes coming out of this was actually reintroducing joins.
Trilogy assumes that joins are expressed at the modeling layer, but when you combine two facts on an uncommon key that you may not have modeled - say, trying to find products that are in the same class from a product dataset, the existing merge syntax was too cumbersome.
We added back query-scoped joins - temporary semantic merges, not literal field-level joins in SQL - and this made a few common cases feel less like the language was fighting you.
Other changes were for convenience; Trilogy supports nested access - order.customer.first_name and namespacing a CTE, producing cte.order.customer.first_name. For non-transformed columns, agents would default to cte.first_name instead of the full path. Even with a helpful error pointing them to the right path, agents would flail or pivot approaches when they just needed a longer access string.
We made it so cte.first_name resolves when unambiguous; this shorthand ended up being very convenient for humans too. Nobody - even machines - likes typing a lot of redundant text, it appears.
Trilogy +10%
Scoreboard: Trilogy 80%, SQL 80%
Step 4: TPC-DS, What the Hell
The remaining 20% gap was flipping from run to run - we went back again to critically review all our evals/questions, tighten up correctness/scoring, and fix more degenerate cases.
Scoreboard: Trilogy 90%, SQL 90%
Lesson learned: we didn't learn our lesson the first time.
At this point, we could stop and measure a Pareto frontier for the first time.
Given that both languages had the same correctness outcome, when should you use Trilogy?
The primary benefit is that with the models defined, the token cost of a question is a static invariant regardless of the size of your DB. So as the number of tables increased, Trilogy would eventually dominate SQL with a schema dump or raw SQL with probes. The slope of this crossover would be determined by how confusing your DB was; if tables were clearly irrelevant, agents would discard them with less discovery work.
The one clear win from the semantic model was reduced DB round trips. If DB access is disallowed, Trilogy is a clear winner.
But this wasn't good enough - for Trilogy to be what we wanted, it needs to clearly beat SQL at any scale.
Step 5: Token Reduction
Trilogy token usage ran ~2-4x raw SQL usage - could we drop that?
Some key wins:
- Prompt optimization. We need to teach a model the language from scratch; there is a minimum viable token penalty.
- Progressive disclosure. We could build tools to discover detailed features on demand, and keep them out of the main plot.
- Reduce tool call loops. We had a write script tool; give it an option to run too. Then give it an option to clean itself up after. 3 agent turns -> 1.
Some Other Lessons
Agent self-evals are a trap. (LLM as a judge is... scary). It's a fine place to start, but:
- Agents massively overindex on the case in front of them. Let one tune a tool and you get a mishmash of hyper-specialized guidance bolted on for one query it just saw.
- They are poor diagnosticians. Agents can do reasonable trajectory analysis under close supervision, but accepting their default root-cause analysis without manually confirming is a recipe for sadness. They tend to speculate without confirming.
- They are not objective oracles. Claude, for instance, tends to assume a task failure means the model underneath is bad (it was quite sure DeepSeek was the problem, as an inferior model), rather than investigating what ended up being very clear framework issues.
Agent Volatility
Models are not interchangeable. We ran most of this with DeepSeek v4 flash - thinking - a hilariously affordable model - and we saw a large tendency to second-guess and validate.
GPT-Luna was much more willing to commit based on the semantic model.
Cheating?
We tried to hide that our eval was based on TPC-DS from agents to avoid them 'cheating' on answers. Surprisingly, mangling table names + context + questions did not degrade the SQL leg; either agents saw through our ruse or pre-training wasn't significantly hurting outcomes.
Where does this leave us?
Getting to parity with SQL on a constrained test set for accuracy, with a token penalty, is table stakes.
Where can we do better? Larger warehouses, where we have aggregates, messy tables, warehouse scale - where searching for the needle in the haystack for every question isn't viable.
Let's try to test that:
Let's get Messy
We built a messy warehouse: we started with TPC-DS's ordinary 42 tables, scaled it to as many as 600, renamed columns, shifted the benchmark parameters, and planted plausible traps like stale _v2 tables, _bak copies, staging tables, and near-correct rollups. We expected agents given just the DB to struggle a bit here; in practice they did not. Across the table-count sweep, bare-discovery agents still answered 18-19 of 20 questions correctly. A modern model with DB access can profile and query its way to a rich semantic understanding, even without provided context.
This shifted the "work" from the context layer to queries. Pure SQL discovery used fewer tokens because it had minimal upfront context, but it treated the database as its schema document, issuing roughly 22 queries per business question.
Injecting the full schema cut that to 7-9 queries, but made context cost grow with the warehouse: from 41k cache-adjusted tokens per question at 72 tables to 129k at 600.
Trilogy operated in a third space - the semantic model handled the exploration work, leaving roughly four to five database executions per question for near-final validation, but at the cost of high language context overhead. This overhead, however, is static at any scale.
| Agent path | Correct answers | Cache-adjusted tokens / question | DB queries / question |
|---|---|---|---|
| Bare discovery | 18-19 / 20 | 42k-50k | 21.8-22.0 |
| Full-schema injection | 16-18 / 20 | 36k-129k | 7.3-8.7 |
| Enriched Trilogy | 18-19 / 20 | 60k-72k | 4.0-5.4 |
The cache-adjusted token calculation weights cached prompt input at 0.1x; this is based on DeepSeek pricing.
At 600 tables, full-schema injection is clearly bleeding token cost; it used 128.8k adjusted tokens and 8.7 database queries per question for 18 correct answers. Much of the injected context is irrelevant. Raw SQL can then minimize the model bill at 49.8k tokens per question because it can prune and progressively explore only what it believes to be in scope. The two enriched replicates used 59.8k-71.6k tokens and 4.0-5.4 queries for 18-19 correct answers. We can't save tokens with the semantic layer because of the cost of the language overhead, but we can guarantee fewer warehouse round trips and a flatter cost as the warehouse grows.
Measurement note
Broader replicate history has swung from 16 to 20 passes, so small accuracy gaps should be read as directional noise rather than a precise ranking. The probe and token separations are more consistent.
So what's best? All these tests ran on DuckDB, where queries are "free" and take milliseconds - it's quite hard to go wrong, and minimizing tokens is pretty reasonable. Raw DuckDB SQL will take you a long way. On a cloud warehouse, every exploratory query acquires a latency floor and a cost; here the tradeoff between tokens and compute gets more interesting.
Lessons
Any measure is a proxy for your end-user experience; optimizing for the wrong measure will optimize for the wrong users. We're not mad that we started with TPC-DS - it taught us a lot - but it probably isn't where we'd recommend someone else start with agentic evals.
Tips
The messy-warehouse waves above are an example of attempting to build a more realistic warehouse for agentic evals. Hex has some fantastic descriptions of why they also needed to build one - we hope to adopt the industry standard.