I get asked the same question in almost every client meeting: which model should we use? I usually answer with a question: which step of your workflow are you worried about? This time I wanted data instead of a question, so I took one real workflow and ran it through seven agents.1
The workflow is prior-authorization intake. An agent opens the inbox, classifies a faxed referral, extracts fields, finds the patient, checks eligibility, applies the payer's rule, fills the portal and submits. Eight steps. A skilled coordinator does it in about eleven minutes.
The setup
Every agent got the same tools, the same system prompt and a 60-step limit. The portal is a sandbox clone that resets between runs, because the last time we skipped that step we had to retract an experiment.2
Where each agent failed
This is the whole note in one picture. Each cell is the share of 20 runs that first failed at that step. Hover a cell for the details.
Model choice mattered less than I expected. Two steps mattered more than everything else combined.
Break 1: the payer rule
Every agent failed the payer-rule step at least once. The pattern was the same: the rule had changed in the synthetic payer's policy document three months earlier, and the older version was still in the referral packet. Agents that read the whole packet sometimes applied the older rule.3
This is not a model problem. It is a data problem that a model exposes. The fix was boring: the rule tool now returns the rule and its effective date, and the check refuses any submission that cites a rule older than the active one.
Break 2: the date picker
The sandbox portal copies a real payer portal, including its date picker, which is clearly older than some of the models. Agents that typed a date into the field often triggered a format error. The open-source browser agent clicked the wrong month in 4 of 20 runs.
The fix was to stop letting agents type dates at all. A dedicated tool sets the date through the portal's own form value, with a strict schema:
{
"name": "portal.set_date",
"description": "Set a date field. Never type into the field.",
"input_schema": {
"type": "object",
"properties": {
"field": { "enum": ["date_of_service", "date_of_birth"] },
"date": { "type": "string", "pattern": "^\\d{4}-\\d{2}-\\d{2}$" }
},
"required": ["field", "date"]
}
}
What did not break
Opening the inbox, classifying documents and submitting were close to perfect for every hosted model. The local models were slower but not worse on those steps. If you are choosing a model for classification alone, choose on cost and data location.4
What I would deploy
- Fix the payer-rule data and the date tool first. That alone moved every agent up by 3 to 5 runs out of 20.
- Then choose the model. On this workflow, Claude and GPT-6 are close; FLX-0142 has the proper 50-run comparison.
- Keep a person on referrals with handwriting. No agent handled them reliably.