Back to Field Notes

The golden nightly's first real runs found three bugs

We built a nightly job that assembles candidate server images whenever anything upstream changes. Its first real runs found three bugs that no test had: a parser that only understood JSON, an error sent to /dev/null, and a promote rule that could never pass.

Covers
Written
Repositories
infinary.io
Reading time
3 min
#golden-image#ci#google-cloud#testing
Contents

What the nightly does

Every Infinary site runs from a “golden” image: Frappe, ERPNext and our own apps, pinned to exact versions and built and tested together. We wanted to know the morning after something changed upstream, a Frappe patch release or a new version of one of our apps, without anyone having to go and look.

So on October 19 we shipped the golden nightly. It’s a Cloud Run job, written against Python’s standard library only, that runs every night and does four things:

  1. checks the newest compatible versions of every app;
  2. if anything changed, builds a candidate image into a separate registry;
  3. validates it on a fresh site;
  4. opens a pull request describing it.

It never pins, approves or rolls anything out. It can’t move the real image tag at all: it builds as its own identity, which can only write candidates. A person approves a candidate, and promoting it is a separate build.

It had 64 unit tests on the day it shipped. Its first real runs still found three bugs.

Bug 1: a parser that only understood JSON

Validation runs a few checks on the fresh site through bench execute: that the site answers ping, and that its PDF engine is Chrome. Our parser read the last line of output as JSON. But bench execute prints a dict as JSON and a plain string bare: pong, not "pong". So the parser returned nothing for exactly the two checks that return strings, and validation could never pass.

The fix falls back to the trimmed text when the output isn’t JSON. We tested it against the shapes bench actually prints: a bare string, a JSON dict, and empty output.

Bug 2: an error sent to /dev/null

The first real nightly build ran at 06:00 UTC on October 20. The image build itself succeeded, then the job exited with no message at all.

The script looked up the new image’s digest with gcloud artifacts docker images describe. For images in Artifact Registry, describe also reads Container Analysis metadata. That needs a permission our job’s identity doesn’t have, and shouldn’t. Our own logins have it, which is why the same command worked when we tried it by hand.

It failed silently for two reasons that compound:

Terminal window
GOLDEN_DIGEST="$(digest_of …)" # digest_of sent stderr to /dev/null
… || die "…" # never reached: set -e exits on the line above

Under set -e, a failing assignment from a command substitution exits on the spot, before the || die on the next line can say anything. The pull request’s failure reason was no help either: it quoted the last lines of output, which were gcloud’s upload progress.

The fix reads digests from the registry’s tag list, which needs only the read permission the job already had. Every lookup now fails loudly with gcloud’s own error, and the pull request quotes our script’s error lines instead of upload chatter. No permission change was needed.

The same run’s log showed a second problem. The planner counted an app pinned to a branch, rather than a release tag, as version 0. Helpdesk requires that app at version 0.0.1 or later, so on any newer ERPNext base the planner would have rejected the upgrade. It now uses the version we recorded for the app.

Bug 3: a promote rule that could never pass

Promoting a candidate copies it into the real registry. The promote build refused any candidate whose “golden id”, the name built from its versions, already existed there on different bits. That sounds safe. But a golden id names versions, not bits, and fetching the same Frappe apps twice doesn’t produce identical bits. The nightly always builds its own fresh base, so for a candidate that changed only one of our apps, the base’s golden id already existed, on other bits, every time. Every such candidate was unpromotable.

The fix copies the candidate’s base under its own nightly tag, named from its digest, instead of reusing the golden id.

What we’d tell someone else

A pipeline isn’t tested until it has run for real, against real permissions and real output. All three bugs passed our unit tests, because each lived at the edge with something we’d stubbed: bench’s output format, the cloud’s permissions, and the registry’s contents.

And never send stderr to /dev/null inside a $( ) under set -e. You’ve built a script that fails without a word.