hohne.meWelcome Hohne

Writing

Gate the agent on a number

A demo shows that an agent can do a task. Deployment needs an answer to a different question, which is when the system is allowed to act without a person checking. I have found that question easier to answer when a small decision model makes the first call and returns a probability, rules in code set the limits, and a person reviews whatever falls below a threshold. This page walks through one test of that design, on a pile of PDFs, and what it taught me about thresholds and reference labels.

Models that answer instead of write

Most language models return text. A decision model takes a question with a fixed set of options and returns a probability for each option. That has practical consequences for anyone putting an agent into production:

On Evaluation I describe a related problem with chat models. On document triage, where the answer must be one label from a fixed set, small chat models often returned explanations, labels that were close but wrong, or extra text instead of the label. A decision endpoint removes that failure, because the model can only choose among the options it was given.

The test

I tested clef-flash, a 9 billion parameter vision decision model from Cloudflare, served through Ollama. The full report is on a11y.world as A small decision model sorts documents from page one. The short version follows.

The set was 75 public PDFs from one organization's websites. The model saw page one only, rendered at 150 dots per inch, with the filename withheld so it could not read the answer off it. It answered one question with nine options for the kind of document, each option with a one line definition, and three yes or no questions. One of those asked whether the page was a scan. The 27 billion parameter model in the same family was tried first and dropped, because it was slower and heavier and the 9B model was accurate enough.

Measure Result
Document kind, against the corrected reference 58 of 60, 96.7 percent
Document kind, against the labels as first written 49 of 63, 77.8 percent
Scanned page detection, against a rule with no model in it 67 of 68, 98.5 percent
Audience and format questions, where a label existed 13 of 18, 72.2 percent
Model errors or empty answers 0
Warm response time per page about 0.6 seconds
Memory while loaded about 14 GB
Same answers on a second run yes

Seven files never downloaded, which leaves 68 answered documents. Five of those had no kind label, so the document kind scores use at most 63.

The reference was the weak part

The first score was 77.8 percent. The labels came from a content catalog written for people browsing a site, not for testing a model, and the model's answer was often more specific than the label. It said "form" where the catalog said "general document."

So I had the 14 disagreements checked again. Three larger models each saw the page image, two pages of text, and the catalog label, but not the small model's answer. They voted on whether the document was clearly one kind. Eleven were, and the majority kind became the reference. Three were not clearly any one kind and left the scoring. On 2 of the 11, the panel's kind matched the catalog label, so the catalog survived the review on 2 of the 14 disputes.

Against the corrected reference, the model agreed on 58 of 60. It agreed on all 49 documents nobody disputed, and on 9 of the 11 that were relabeled. The two remaining misses both sit on a boundary between kinds, where a person could argue either way.

For anyone evaluating an agent, this is the result I would take away. Before you blame the model, read every disagreement, because the answer key can be wrong. A panel of models is a repeatable second opinion, but it is still not a person, and I froze its labels so later runs compare against the same file.

Where to put the threshold

The probability the model gives its chosen answer is the number you would route on, so I grouped the 68 answers by it.

Probability of the chosen kind Answers Correct, of scored
0.30 to 0.49 15 10 of 10
0.50 to 0.69 17 13 of 15
0.70 to 0.89 23 22 of 22
0.90 to 1.00 13 13 of 13

In this sample the least confident answers were all right, and both misses fell between 0.50 and 0.69. A threshold at 0.70 would have caught both misses, and it would also have sent 32 of the 68 answers, almost half, to a person. That is the trade a product team has to make on purpose. Treat the probability as a reason to look at an item. It does not guarantee the answer, and the threshold has to come from your own data.

The scan question went better. A rule with no model in it, fewer than 25 characters of text on page one plus a single image covering most of the page, agreed with the model on 67 of 68 documents. The one miss was the only fully scanned PDF in the set, which the model scored at 0.07 for "scanned." A threshold would have sent it for review. Having a rule with no model in it beside the model gave me a check that does not share the model's blind spots.

The pattern I would ship

  1. A small decision model makes the first call and returns probabilities.
  2. Rules in code decide what is allowed. A model can propose a disposition, but it cannot override a rule.
  3. Anything under a threshold goes to a person, and the threshold comes from measured disagreements.
  4. Log every decision with its probabilities and the rule that fired. That log is an audit trail you can count.

This is how the Document Disposition Workbench on Projects works. It sorts each document into update, convert, or retire with rules that content owners can read and edit, and then the owners review and override the recommendations. My model evaluations point to the same division of labor between small models, larger models, and people. A Qwen model with 0.8 billion parameters passed every case in my classification and structured extraction categories, and answered in under a second, which makes it a good first step in front of larger models. At the other end, some design judgment tasks defeated every model I tested, including the largest. The clearest was noticing that a heading is an image of text and should be rebuilt as real text. Tasks like that stay with a person.

I use the same split in an accessibility MCP server. Rules in one file decide each disposition. No model call decides one. Every output lists the rules that fired.

Limits

The model saw page one only, and a long document can change kind after its cover. The corrected reference came from a model panel, not from people, and the panel reviewed only the 14 disagreements. Seventy five documents from one organization is a small sample. The result shows that a small decision model is worth testing on your own documents, not that it will score the same there.

Try it on your own documents

Take 50 of your own PDFs, write down the kind you believe each one is, and run the same question. Score against your labels, then read every disagreement before you trust either side. The agent test corpus on a11y.world ships with a scorer that reports precision and recall per rule and lists each miss, which is a good shape to copy.

Sources