Back to Field Notes

A control plane in three days, and nothing protecting it

In one week of June we went from a spec to a deployed control plane, a signed-in customer portal and an agent on our own ERPNext site, with no automated tests. The review that brought the first tests, on July 2, also found that "Schedule tonight, 2 AM" ran immediately.

Covers
Written
Repositories
infinary.io, infinary_agent
Reading time
3 min
#control-plane#portal#testing#ci
Contents
pull requests, June 22–26
43
automated tests until July 2
0
control-plane tests on July 2
40
agent tests on July 2
37

From spec to running in an afternoon

At 11:31 on Tuesday, June 23, a pull request merged with a note that the self-serve flow, the agent on each instance and the upgrade prompts were “spec only — not built”. By that evening there was a deployed control plane, a customer portal you could sign in to, and an agent installed on our own ERPNext site.

June 23, Denver time
  1. A plan for the customer portal: a control plane on an Ubuntu VM, a SolidJS single-page app on Cloudflare Pages, Firestore and Pub/Sub.
  2. The /v1 API contract, frozen as a Zod schema. It became the one definition both ends compile against, and it outlived the plan.
  3. A stub server in Hono implementing the whole contract against an in-memory store.
  4. Terraform for a single ERPNext instance, which we applied to bring up our own site.
  5. The control plane deployed, on Cloud Run rather than the VM the plan had named three and a half hours earlier.
  6. Google sign-in.
  7. A Firestore-backed store, one organization per Google account, with every read scoped to the caller's organization.
  8. A real outbound agent with a per-instance token, replacing the simulated one.
  9. The agent moves to its own repository; its first version is tagged at 20:15.

The rest of the week kept that pace. Wednesday brought 13 merged pull requests: checkout wiring for Stripe (inert until real keys were set), a provisioning worker that drives the Terraform through Pub/Sub and retries the next zone when one is out of capacity, idempotency keys on every write, an audit log, team roles, an admin console and the first tagged release, v0.9.0. Thursday added a kill switch that can pause one instance or the whole fleet, an auto-update policy, a blue/green self-update for the agent, and backups through Google Cloud Backup and DR. Friday added per-app updates. Between Monday and Friday we merged 34 pull requests in infinary.io and 9 in the new agent repository.

The only ERPNext instance any of it managed was our own site.

What was checking it

Reviews, and hands. Before several merges we ran an adversarial review of the branch. The one on the portal’s first flows raised eight findings, five of them real and in scope, including a blocker: a fresh live-progress connection started one event behind the latest, so stages that fired before the browser connected never appeared. The one on the update engine found 14, including the order of a rollback (restore the database before the code, and never put older code onto a database that has already been migrated forward).

The assessment

On July 2 we stepped back and surveyed both repositories and the planning documents. The summary’s first line:

The risk is not code quality. It is that nothing protects that quality

The same day we added the safety net. CI now runs on every push and pull request: Biome lint, the site build, the portal’s typecheck and build, and the control plane’s typecheck and tests. The control plane got its first suite, 40 tests covering the six upgrade gates, the job queue, the action ledger, the single-flight lock and round-trips through the /v1 contract, including heartbeats from older agents. The agent repository got 37 tests and its own CI, pinning the rules its rollback depends on: never move code backwards onto a migrated database, and always clear maintenance mode.

Two flows that didn’t do what they said

The same review went through the flows we had already demonstrated, and two of them didn’t do what their screens said:

What we’d tell someone else