Decision 01 of 05 · 2 min read
The model plans; the pipeline writes the SQL
Why doesn't the AI write the query?
Gemini reads the question and a curated description of the metric, its statuses and its tables. It returns exactly one of three typed results:
- an intent: by category, a total, one category, or month over month, with a period;
- a clarification question, when the measure or the period is ambiguous;
- a refusal, for a write request or a restricted table.
The pipeline then compiles the one query for that intent, with the period and category as bound parameters. The compiled SQL still goes through the SQL guard, so a compiler mistake cannot slip through unchecked.
It didn't start this way. The first candidate let the model write SQL from trusted examples, and it failed its sealed test by one question. The revision moved SQL into the pipeline and wrote the clarification rules from the metric's contract, and it passed a fresh sealed test. Which change fixed which miss is not known: the used test was never opened again.
The trade-off
Trellis can't answer a question outside its compiled query shapes. That is deliberate: every published number comes from SQL that was reviewed once, not generated fresh for each question.
What was shown
- StagingThe model returns an intent, a clarification or a refusal; the pipeline compiles the SQL, with the period and category as bound parameters.
- Live evalThe first candidate scored 34 of 36 on its sealed test, one short of the bar, with no unsafe answers.
- Live evalThe revised candidate scored 36 of 36 on a fresh sealed test, with no unsafe answers, for an estimated USD 0.217 in model and query cost.
- LimitCustody was one person: an AI model drafted the sealed sets from an agreed plan, and one person reviewed, sealed and ran them. That is weaker independence than a separate custodian.
- LimitA seal proves the files were not changed, not that the expected answers are right.
- LimitBoth sealed sets are now used up; the next independent result needs a new one.
“Unsafe” here means publishing an answer when the expected outcome was a refusal, a question back, no answer or unavailable; an answer with wrong values counts against accuracy instead.