Public engineering laboratory · since 2025
Models, agents and tools, run against real healthcare workflows on synthetic data. Every experiment has a question, a hypothesis written before the first run, a published protocol and raw logs. Failures stay online.
Each card shows the question, the status and the result in the form that fits the experiment: lanes for battles, run strips for reliability, thresholds for failures.
Hypothesis: three agents in parallel clear a 200-item queue 2x faster than GPT-6 alone.
Hypothesis: planner, extractor and verifier fit in memory at once with 4 concurrent agents.
Result: Qwen 32B at Q8 gives the best accuracy per second. The 72B model adds 1.8 points at 2.6x the time.
Result: GPT-6 fastest by median. Claude most reliable. Qwen local 13x cheaper per run.
Result: 97.3% success. 21 of 27 failures hit 3 payer portals during their 06:00 to 07:00 maintenance window.
Result: the agent was 4.1x faster; specialists scored 6 points higher. Agent draft plus specialist review scored highest.
Result: use the API where one exists. Agents beat RPA only on portals that change more than once a quarter.
Result: useful for scaffolding. The unsafe bot led to a new guardrail: no submit action without a validation step.
Why inconclusive: 32 of 50 samples excluded for image quality. Re-running with a new synthetic set.
What we keep: the call-flow state machine, now in production with a hosted model.
Why it stays online: so anyone who cited it can see what changed.
So far: 97.1% success, failures again clustered in portal maintenance windows.
| ID | Question | Type | Status | Tested | n | Cost | Date |
|---|---|---|---|---|---|---|---|
| FLX-0149 | Three concurrent agents on one prior-auth queue | Agent battle | Running | GPT-6, Claude, Qwen 32B | 31/50 | ~$140 | 09-18 |
| FLX-0148 | Six local models on 256 GB unified memory | Local benchmark | Running | 6 local models | 212/300 | $0 | 09-19 |
| FLX-0147 | Does FLX-0140 hold in a different week? | Reliability | Replicating | Claude | 684/1000 | ~$150 | 09-15 |
| FLX-0145 | Local models on handwritten requisitions | Local benchmark | Concluded | 6 local models | 400 | $0 | 09-05 |
| FLX-0142 | GPT-6 vs Claude vs local Qwen on prior-auth intake | Agent battle | Concluded | GPT-6, Claude, Qwen 32B | 150 | $58.40 | 09-12 |
| FLX-0140 | 1,000 runs of one denial-appeal agent | Reliability | Concluded | Claude | 1000 | $212 | 08-29 |
| FLX-0138 | Four billing specialists vs one agent on appeals | AI vs human | Concluded | Claude, 4 people | 200 | $96 | 08-22 |
| FLX-0136 | Cost per 1,000 claim status checks | Cost | Concluded | API, RPA, agent, human | 4000 | $44 | 08-15 |
| FLX-0133 | Vision model vs OCR on fax cover sheets | Tool test | Inconclusive | Vision model, OCR | 18/50 | $6 | 08-08 |
| FLX-0131 | Fully local voice agent for rescheduling | Voice agent | Failed | Local STT + 32B | 40 | $12 | 08-01 |
| FLX-0129 | Browser agents on payer portals, round 1 | Agent battle | Retracted | 3 browser agents | 90 | $27 | 07-25 |
| FLX-0127 | Can an agent write its own RPA bot? | Weird idea | Concluded | Claude, coding agent | 12 | $31 | 07-18 |
A failed experiment is a result. Most of them leave behind a component, a guardrail or a threshold that now protects a production system.
Callers talked over the agent in 31 of 40 test calls.
The call-flow state machine. It now runs in production with a hosted model and passes the same test at 0.7 s.
Phone-tree speech recognition misheard member IDs in most calls.
A keypad-only path for two payers whose phone trees accept tone input. Used in production since July.
Only 18 of 50 samples were usable after image-quality exclusions. Too small to call.
FLX-0151 is proposed with a new synthetic set generated at real fax resolution.
ProposedThe sandbox portal cached answers between runs, so later runs were easier. The reported 98% was not real.
The harness now resets the sandbox between every run and checks for cached state. FLX-0142 replaced this experiment.
The hypothesis and the thresholds are written down before the first run. That is the single rule that keeps a lab honest.
Plain language, tied to a real workflow a client runs.
And pass thresholds, written before the first run.
Dataset, environment and harness version frozen.
At least 50 per contender, in random order.
A second run, another week, another engineer.
Result, raw logs and harness. Failures too.
| Production candidate threshold | Minimum |
|---|---|
| End-to-end completion | ≥ 95% |
| Field-level accuracy | ≥ 98% |
| p90 time vs median | ≤ 1.6x |
| Unsafe actions (submit without validation, wrong patient) | 0 |
| Cost per case vs human baseline | < 25% |
| Replication difference | ≤ 2 points |
# Reproduce any Lab experiment on your own hardware git clone https://github.com/flobotics/lab-harness cd lab-harness && make env flx run FLX-0142 --runs 50 --replicate