- our estimate for a run
- 5–10 min
- the first real run
- 1 min 52 s
- control-plane tests in it
- 295
- failures and errors on Kremio Connect main
- 18 + 1
What stopped
As of this writing, October 21, it’s still not running jobs. Orca-strator lives under a different GitHub account and kept its CI.
Three things stopped with it: the website repository’s CI (six jobs), the workflow that builds and publishes our golden images, and the portal’s deploy. The timing was bad. Changes across five repositories that replace Frappe Assistant Core with our own MCP server were about to merge, and their golden image build was refused like everything else.
- GitHub Actions stops starting jobs for the organization.
- Temporary rule: merge once the tests pass locally, with the counts in the pull request.
- Standing rule: don't rely on GitHub CI; test locally, use Cloud Build for builds and heavier runs.
- The first CI run on Cloud Build, with Nikolas's go. All six jobs pass in 1 minute 52 seconds.
Merging on local test runs
At about 11:10 Nikolas set a temporary rule: merge once the tests pass locally, with the commands and counts in the pull request. Anything that touches the golden image, release tags or production still needs his yes. Each repository got a local stand-in for its CI: ruff, licence headers and the (Frappe's command-line tool, and the folder of apps and sites it manages. It installs apps, builds assets, migrates sites and runs tests.) suite in a dev container for Kremio Connect; ruff, two mypy runs and pytest for the sidecar; npm ci, Biome, the typechecks, the control-plane tests, the catalog check and the bring-up scripts for the website repository. Infinary Controls had never had CI, so it got pytest. Golden images got a Cloud Build path the same day.
By 15:05 the rule was standing: don’t wait for, re-run or debug GitHub CI. Run the tests locally, and use Cloud Build for builds and heavier runs.
CI on Cloud Build
cloudbuild.ci.yaml runs the same six jobs as the Actions workflow, on node:22-bookworm, on an E2_HIGHCPU_8 machine with a 30-minute timeout:
- one
npm ci, then the jobs run side by side on the shared workspace; - each job records its output and exit code instead of failing the build, so one run reports every failure, the way separate Actions jobs do, and inside a job a failing command stops that job;
- a last
summarystep lists each job’s result, the test counts and Biome’s totals, then fails the build if any job failed.
scripts/cloudbuild-ci.sh <ref> [--dry-run] [--yes] submits one commit. It exports the commit with git archive, so only committed files go up, with no node_modules, no uncommitted edits and line endings as stored. It prints the plan, asks, submits, and reads the summary back from Cloud Logging for pasting into the pull request. There’s no trigger and nothing reports back to GitHub; that would need Cloud Build’s GitHub app and a separate decision.
The pull request estimated a run at “roughly 5 to 10 minutes”, and said it was an estimate, not a measurement. The first real run, at 15:13 with Nikolas’s go, took 1 minute 52 seconds from creation to finish, and about 50 seconds once the source was uploaded. npm ci took about 29 s. All six jobs passed, including 295 control-plane tests.
That first summary had one bug. Its test-count line was empty, because without a terminal node --test prints TAP (# tests 295) rather than the format we matched. And gcloud builds submit doesn’t stream logs that go only to Cloud Logging, so the script now takes the build id from gcloud’s output and reads the summary back itself.
Rolling it out
On Sunday night the same pattern went to three more repositories: Infinary Controls (pytest), the sidecar (ruff, two mypy runs, pytest) and Kremio Connect. Kremio Connect needs a whole Frappe v16 bench, so its build starts MariaDB 11.8 and two Redis containers on Cloud Build’s own network, runs bench init, adds ERPNext, creates a site and runs both apps’ tests. Runs start from a clean clone, because Windows checkouts have CRLF line endings.
The first real runs came early Monday. Controls passed 518 tests with 1 skipped, the sidecar 322.
- A setup needed HRMS. A test class created a user of HRMS’s Employee Self Service type, and the CI bench has no HRMS.
- The site’s URL depended on who asked. 17 tests were refused because the URL the test process saw (from the site’s name) differed from the one the test request saw: Werkzeug’s test client always sends
Host: localhost. With ahost_name, both sides agree. - One test depended on module order. Frappe walks test modules in unsorted filesystem order, and Werkzeug’s test client keeps an emptied
sidcookie that Frappe then treats as an expired session, so the test got a 401 instead of a 403. On GitHub it ran 28th of 52 test classes; on our machine, 45th.
With the test fixes merged alongside the config, the combined run had 376 tests and one error, a sandbox memory limit, which had its own fix that night.