- from first merge to the realtime fix
- ~38 h
- infinary_controls on Sunday alone
- v0.2.0 → v0.4.1
- Vertex models that had ever worked in the image
- 0
- fallback we built for a connection that never connected
- 12 s
What Aria is
Aria is a chat assistant that lives inside the customer’s ERPNext: a page in the Frappe desk, and a drawer you can open from every desk page. It answers questions about the ERP, can hand bigger jobs to the hosted agents, and can propose a change for a person to approve. It’s built on Orca-strator’s chat machinery, and by default its orchestrator runs as a container next to ERPNext on the customer’s own VM. A customer without a VM can have the same container as a separate service of their own; one orchestrator shared between customers was rejected, as a path from one customer’s data to another’s.
In August the only ERP it ran on was our own.
The timeline
- The orchestrator, the ERP side and the hub side merge.
- A startup bug in the image is fixed: it built fine and could not start.
- The chat drawer on every desk page; per-user features follow all day.
- We find no Vertex model had ever worked in the image, and add the missing SDK.
- Gemini calls go to the Vertex global endpoint.
- Reasoning shown, list numbering fixed, and a 12-second fallback for missing live replies.
- The post-deploy smoke test stops crying wolf.
- The realtime Origin fix is baked into our image.
The model is chosen per client, with a built-in fallback chain of Gemini on Vertex AI, then Claude, then OpenAI. On a Google Cloud VM, Vertex authenticates as the VM’s own service account, so there is no API key to store. Replies stream from the orchestrator to the ERP app, which relays them to the user’s browser over Frappe’s realtime channel.
The startup bug at 20:28 was one the hosted agent runner had already hit five days earlier: the image’s entrypoint named a script our build never installs, so it built fine and couldn’t start. The runner’s fix had never reached this image.
Sunday, August 30, from 01:07 onwards, added the drawer on every desk page, per-user chat history, a per-user model choice from an allow-list the organization sets, a choice of tone, awareness of the page the user is on, approval cards, a control for how hard the model thinks, a redesign, and an opt-in badge for items needing attention. infinary_controls went from v0.2.0 to v0.4.1 by the end of the day.
The rules it’s built on
- Aria only proposes. Its
propose_actiontool hands a proposal back for display and does nothing else. What happens next is up to a person: their click calls an ERP method that runs as that person, refuses our agent account, requires a draft document that the person has permission to submit, and never bypasses permissions. Separately, Aria’s own tool gate refuses submit, update, delete and cancel tools even if a broad allow-list would let them through. - Text from the browser never becomes instructions. The tone is a key into a fixed list on the server, which owns the actual wording. The page the user is on (its route, doctype and document name) is cleaned and wrapped in the prompt as data, not as an instruction. Reasoning effort is a fixed set of values on the wire. A review of the tone feature found one more field that reached the system prompt unchecked; it now accepts only known agent IDs.
No Vertex model had ever worked
The fix was one dependency. The lesson was a post-deploy smoke test that makes one real model call after every deploy. Its first version cried wolf twice: it called the region our VM is pinned to, and the newest Gemini models, including the -latest aliases the box uses by default, are only served from Vertex’s global endpoint. Aria had been getting the same 404s; at 18:16 it started sending Gemini calls to the global endpoint, and the smoke test was fixed to match that night. Claude on Vertex was wired up the same day but still waiting on access to the model, so on our site Aria’s default model was Gemini.
Durable history for the central service
The separate per-customer service runs on Cloud Run, where an instance can disappear at any time. Aria’s history lives in SQLite behind about 5,200 lines of hand-written store code, so rather than port it to Postgres we replicate the file to Cloud Storage with Litestream and restore it on boot. If the restore fails, the service refuses to start rather than starting empty and overwriting the customer’s history. A real test deployment found three bugs no review had: Cloud Run’s front end rejected every request before it reached the container, the app’s host allow-list refused Cloud Run’s hostname, and deletion protection blocked teardown. After the fixes, a conversation written on one instance came back on a fresh one.
The header that kept it silent
At 21:58 on Sunday we shipped fixes for three problems reported from the live chat: Aria’s reasoning was never shown, ordered lists all numbered “1.”, and replies only appeared after refreshing the page. We treated the last one as a flaky connection, and added a fallback that fetches the saved answer if nothing arrives within 12 seconds.
We patched nginx on the box first, then at 08:53 on Monday, August 31, baked the fix into our image: Origin now uses the request’s own host, and the build fails if the upstream template line ever moves.
proxy_set_header Origin $proxy_x_forwarded_proto://${FRAPPE_SITE_NAME_HEADER};proxy_set_header Origin $proxy_x_forwarded_proto://$host;It was the same error message as on June 23, when we brought the site up, with a different cause.