Why these surfaced now
Since October 16 our CI runs on Cloud Build. Its configuration runs six jobs side by side on one machine: lint, the site’s tests and build, the bring-up script tests, the portal, the control plane, and the catalog checks. That’s efficient, and it means any test that measures time is measuring a busy machine. Within two days, three control-plane tests failed on pull requests that hadn’t touched them.
1. A performance test with an absolute bound
The test timed one parse of 500 synthetic instance records and failed at 100 ms or more. Locally it took about 12 ms. On Cloud Build, on October 19, it took 123.6 ms, and failed a PR that didn’t touch that code. We reproduced it locally by oversubscribing the CPUs: single runs reached 105–170 ms.
The easy fixes don’t hold:
- Taking the best of several runs survives a few unlucky runs, but not a runner that is slow for the whole file.
- Raising the bound loses the point of the test. The old 100 ms limit only just caught a deliberate tenfold slowdown, at 101–107 ms.
So the test now measures against a baseline taken in the same process. Each of seven rounds times a plain JSON round-trip of the same 500 records, then the parse. The assertion is that the best parse time is under five times the best baseline time. A slow machine slows both equally.
| Change to the parse | Ratio to baseline | Result |
|---|---|---|
| none, quiet machine | 1.2–1.3 | passes |
| none, 80 busy processes on 24 cores | 0.8–1.6 | passes |
| made 5× slower | 5.7–6.1 | fails |
| made 10× slower | 12–13 | fails |
It’s a guard against an order-of-magnitude regression, and it says so in its comment.
2. A test that read the real clock
A test checked that an app request stays approved “outside 02:00–03:00 UTC”, the maintenance window when approved requests are dispatched. It got the current time from the real clock. On the evening of October 19, a CI run happened to start at about 02:07 UTC, inside the window: the request was dispatched, and the test failed on a docs-only PR.
The code under test already took the time as a parameter, so the fix was in the test: it now sweeps at a fixed 01:59 UTC, and the unknown-timezone case at 02:30. We checked every other control-plane test for Date.now() or a bare new Date(); none had hour-of-day logic.
3. A streaming test paced by the clock
Our model gateway relays a streamed reply event by event, and a test checked that it doesn’t buffer. A mock model service sent events 25 ms apart, and the test checked that they arrived spread out the same way. It failed once on Cloud Build and passed on a re-run.
We ran a copy that recorded timings instead of asserting, 100 times, with one busy-loop process per logical core (24 on that machine). The typical event took about 0.9 ms to cross, and 5 ms at the 95th percentile. But 6 of the 200 stream legs (3%) had one event arrive 25–47 ms late, past the 25 ms gap. Widening the margin would only move the threshold, because a stall can be as long as it likes.
So the test no longer uses the clock to pace the stream. The mock now holds each next event until the test confirms the client received the previous one, with a 2-second limit, and the test asserts that every wait ended with a confirmation. A relay that buffers never sends that confirmation, so the test still fails against two deliberately buffering relays. A slow machine only makes the test slower. Under load it passed 100 of 100 runs at 24 busy processes and 60 of 60 at 48. It also runs in about 70 ms instead of about 400.
The rule
All three fixes changed only tests. None of the three failures was a bug in the product, but each cost a red PR and a re-run, and a flaky test teaches people to ignore red.