- bugs that read zero for unknown, in two days
- 4
- price-map entries that really are free
- 130
- ERP tools a hosted lane was given
- 9–10
- tool calls before it reported success
- 0
Zero is a claim
Orca-strator, our visual builder for AI workflows, refuses a run whose expected cost exceeds its budget and stops starting new lanes once the budget is spent. That only works if the costs are right. In its first two days, four separate bugs made them read zero.
Tokens, read from the wrong place (August 14). The first version read token usage from chunk.usage on each streamed chunk. LiteLLM only fills that in when the request asks for usage in the stream, and ours never did, so token counts would have been zero on every real provider run.
Prices for models it had never heard of (August 15). We checked LiteLLM against our own pinned version and its bundled price map. For model names it didn’t know, it raised an error, which our code correctly turned into “unknown”. Except for Anthropic-shaped names: for those it returned a price of zero. So a model released after the price map was pinned (exactly the case free-text model names exist for) would run, bill real money, and book $0.00 as a known cost. The estimate would read zero too, so the budget check passed going in and could never fire on the way through. We didn’t fix it by treating every zero as unknown: 130 entries in the map are genuinely free, from local models to free tiers. Whether a model is in the price table at all is what separates “known to be free” from “not known”.
Usage, one chunk too late (August 15). Fixing where we read usage wasn’t enough. On both OpenAI and Anthropic, usage arrives on a chunk after the one that carries the finish reason, and our stream handler emitted its “finished” event as soon as it saw the stop. Every real lane still reported no counts. We found it by standing up a loopback server that speaks each provider’s real streaming format, with no network and no key. The same change made a transport error, a timeout or a cancellation book “unknown” instead of a confident $0.00.
The planner nobody counted (August 15). A planner that splits a request into tasks makes a real, billable model call, and nothing recorded it. We measured it on a three-lane flow: the ledger’s total left out an entire model call, and the ledger still marked itself complete.
The planner’s cost is now booked before the fan-out, so the budget can act on it, and the budget can refuse a planner as well as a lane. Each new test was run against the code without its fix to prove it fails.
On August 15 the rule went into the project’s handoff document:
Unknown is never zero. A cost that could not be determined is
Noneand renders as “unknown”. A budget built on a silently-zero cost never fires.
Carried outward
When we built the hosted agent platform on Orca in late August, the rule went with it into each layer. The control plane’s contract for an agent run, written on August 24, carries a spendExact flag: when it’s false, the spend figure is a lower bound, “and anything billing off it has to say so”. The runner reports spend with that flag beside it. The run history in the portal shows a lower bound as a lower bound, and the read-only run mirror in the customer’s own ERP stores the same flag, so a partial figure is never shown there as a total.
The same mistake, with work instead of money
On August 28, with the first hosted agents running against our own ERPNext site, the rule turned out to apply to more than money.
All eight ERP agents now require a tool call on the first step, and a model that can’t be forced to call a tool gets a hard instruction instead, plus an event recording that it couldn’t be forced.
Clean, or never ran? Our code inspection agent checks a site’s server scripts, client scripts and custom fields, and writes its findings into the ERP as a note. Run on a clean site, it succeeded and created nothing, so “audited and found clean” looked exactly like “never ran”. It now always writes its report, and when there are no findings, the report says how many of each it examined and that nothing was found. Published agent versions can’t be changed, so the fix went out as a new version, 1.0.1.