- rebuilds at unchanged tags that day
- 4
- until we reverted the pin
- 4 min
- registry manifests untagged
- 121 of 156
- before untagged images are deleted
- 30 days
Why we were pinning by digest
Every ERPNext instance we manage runs a (The server image every Infinary site runs: Frappe, ERPNext and our own apps, pinned to exact versions and tested together.) we build, and an update is a move from one image to another. The control plane says which image is approved; the agent on the instance compares that with what it’s running, and if they differ it takes a backup, puts the site into maintenance mode, swaps the image and runs bench migrate.
On Tuesday, August 25, we learned that a tag is only a label. Rebuilding an image at unchanged coordinates (the same version numbers, so the same tag) publishes different bits under the same name. That happened to our images four times that day alone, whenever a Dockerfile or workflow changed without a version bump. Two instances reporting the same tag could be running different software, and nothing in the system could tell. The only way we found out was by comparing the registry by hand.
So we started moving every reference to digests: the content hash of the image, which can’t be moved. That afternoon the base ERPNext image we build from was pinned by digest, and the agent started reporting the digest of the image it actually runs.
What went wrong
The next step was to set the approved image to the digest of the image our fleet was already running. Our fleet was our own site.
We reverted the pin four minutes later. Nothing fired. The check we had run before the change proved the fleet was running the right bits; the mechanism compares references. We had verified the wrong invariant.
What we changed
- The agent compares images, not names. Docker already knows when two references resolve to the same local image, so the check needs no registry call and no credentials. It mustn’t over-match either: for an image the instance has never pulled, the answer is “unknown”, so a genuinely new release can never be mistaken for a re-spelling of the current one. Tests cover both directions, and each was run against the code without its fix to prove it fails.
- The control plane compares digests when both sides have one. The agent now reports the digest of what it runs. When either side lacks a digest (an older agent, or a tag the control plane can’t resolve, since it has no Docker), it falls back to comparing strings. Falling back to “different” is the safe direction: offering a redundant update beats hiding a real one.
- Both had to ship before we tried again. The agent and control plane fixes merged within a minute of each other, and the agent’s needed a release to reach the instance.
The retention trap
Two small scripts close it. One pins an image by giving its digest a tag named after the digest itself, which no future build can collide with, and checks the tag stuck. The other reads what the control plane is actually serving, not what we believe is set, and fails if the pinned image is missing or untagged.