- runs examined, Oct 9–20
- 156
- ERP connection failures
- 80
- failures after the first minute
- 0
- runs succeeded after the fix
- 3 of 3
The symptom
Hosted agent runs execute as Cloud Run jobs. Each run’s first request to the customer’s site is a run claim, and from October 19 those claims started timing out. About one run in three failed before doing any work.
The problem wasn’t new, only newly loud. Before the run claim existed, the same failure showed up as a warning (the ERP tool server “is not answering”) and the run carried on without its ERP tools. About 40% of runs had logged that warning since at least September 20, the oldest log we still had. The claim turned a silent degradation into a visible failure.
The diagnosis
We pulled every runner log from October 9 to 20: 156 executions, 80 of which had an ERP connection failure. Then we bucketed the failures by how many seconds into the run they happened:
Show data
| connection failures | |
|---|---|
| 0–9 s | 23 |
| 10–19 s | 26 |
| 20–29 s | 22 |
| 30–39 s | 6 |
| 40–49 s | 2 |
| 50–59 s | 1 |
| later | 0 |
Not one failure came after the first minute, though runs make many ERP calls after that. Only connections that left through Cloud NAT failed; calls to Google’s own services from the same runs went through. The NAT gateway showed no dropped packets and used at most 10 of its 64 ports, and the ERP server was healthy.
That pattern matches something Google documents for Direct VPC egress with Cloud NAT: a new instance can take 30 seconds to a minute or more before outbound connections work. Every agent run is a fresh instance, and the run claim was the first thing each one sent through NAT. We were confident about the start-up effect and less sure of the exact mechanism; we had no packet capture of a failing run.
Fix one: wait for the network, and don’t let one call kill a run
The runner now waits for egress before its first request to the site. It opens a plain TCP connection with a 3-second limit, retrying after 0.5, 1, 2 and 4 seconds and then every 5 seconds, for up to 120 seconds. If the site never answers, the run exits with a new code, site_unreachable, having sent nothing.
Inside that window, the run claim and the tool server’s connect are retried, but only for failures in the connect phase, where nothing can have reached the site. A request that may have arrived is never retried, because a run claim is single-use. And one dropped ERP call now fails that call, not the whole run.
We also stopped a library from fetching a model price list from GitHub at start-up. On October 19 that fetch failed in 16 of 23 runs, and cost about 20 seconds each time.
What we got wrong
The 120-second window was measured from process start: the process’s age, read from /proc/self/stat against the boot clock. Our own diagnosis had recommended that anchor. On Cloud Run, that age over-reports, sometimes by more than the execution has existed. Three runs on the new runner, the same afternoon:
- one run’s gate got about 11 seconds, not 120, before it gave up, because the clock said the window had already passed;
- one logged “egress ready after 242.0 s” for an execution about 154 seconds old;
- one logged “835.0 s” for an execution about 180 seconds old.
There was a second problem with the anchor. Before the gate, the runner fetches its run spec from our control plane, which on one run took 140 seconds. A window counted from process start spends that time too.
Fix two: start the clock where the wait begins
deadline = startup_clock() + STARTUP_WINDOW_SThe deadline is now set when the egress wait begins, and the same deadline bounds the claim and connect retries. The gate is the first thing that goes through NAT; anything before it talks to Google and shouldn’t use up the window. The process-age code is gone, and the logs now report the wait itself.
The tests run on a clock we turn by hand, starting at 950,000 seconds, a process up for eleven days. Under the old code that would have closed the window before it opened. As a mutation check, we anchored the deadline 242 seconds early, as the old code effectively did on one run; three tests fail.
The result
On the fixed runner, 3 of 3 runs succeeded. They waited 35.6, 27.5 and 3.5 seconds for the network, and one tool-server connect needed a retry. Three runs is a small sample, and we’re watching the logged waits. If they turn out too long, the other option is a different egress path for the runner.
What we’d tell someone else
And when you set a timeout, anchor it to the thing you’re waiting for. A process’s age is a different clock, and on a managed platform it can be wrong.