Back to Field Notes

The runner's egress wait, and the clock we anchored it to

About one hosted agent run in three couldn't reach the customer's ERP. Every failure came in the run's first minute. Our first fix measured its 120-second wait from process start, a clock that over-reports on Cloud Run. The second fix starts the clock when the wait begins.

Covers
Written
Repositories
Orca-strator, infinary.io
Reading time
4 min
#hosted-agents#orca#google-cloud#observability
Contents
runs examined, Oct 9–20
156
ERP connection failures
80
failures after the first minute
0
runs succeeded after the fix
3 of 3

The symptom

Hosted agent runs execute as Cloud Run jobs. Each run’s first request to the customer’s site is a run claim, and from October 19 those claims started timing out. About one run in three failed before doing any work.

The problem wasn’t new, only newly loud. Before the run claim existed, the same failure showed up as a warning (the ERP tool server “is not answering”) and the run carried on without its ERP tools. About 40% of runs had logged that warning since at least September 20, the oldest log we still had. The claim turned a silent degradation into a visible failure.

The diagnosis

We pulled every runner log from October 9 to 20: 156 executions, 80 of which had an ERP connection failure. Then we bucketed the failures by how many seconds into the run they happened:

0102030failures0–9 s10–19 s20–29 s30–39 s40–49 s50–59 slaterconnection failures, 0–9 s: 23connection failures, 10–19 s: 26connection failures, 20–29 s: 22connection failures, 30–39 s: 6connection failures, 40–49 s: 2connection failures, 50–59 s: 1connection failures, later: 0
0102030failures0–9 s20–29 s40–49 slaterconnection failures, 0–9 s: 23connection failures, 10–19 s: 26connection failures, 20–29 s: 22connection failures, 30–39 s: 6connection failures, 40–49 s: 2connection failures, 50–59 s: 1connection failures, later: 0
ERP connection failures by seconds into the run, October 9–20 Of 80 connection failures across 156 runs, 71 came in the first 30 seconds, 9 in the next 30, and none after the first minute.
Show data
connection failures
0–9 s23
10–19 s26
20–29 s22
30–39 s6
40–49 s2
50–59 s1
later0

Not one failure came after the first minute, though runs make many ERP calls after that. Only connections that left through Cloud NAT failed; calls to Google’s own services from the same runs went through. The NAT gateway showed no dropped packets and used at most 10 of its 64 ports, and the ERP server was healthy.

That pattern matches something Google documents for Direct VPC egress with Cloud NAT: a new instance can take 30 seconds to a minute or more before outbound connections work. Every agent run is a fresh instance, and the run claim was the first thing each one sent through NAT. We were confident about the start-up effect and less sure of the exact mechanism; we had no packet capture of a failing run.

Fix one: wait for the network, and don’t let one call kill a run

The runner now waits for egress before its first request to the site. It opens a plain TCP connection with a 3-second limit, retrying after 0.5, 1, 2 and 4 seconds and then every 5 seconds, for up to 120 seconds. If the site never answers, the run exits with a new code, site_unreachable, having sent nothing.

Inside that window, the run claim and the tool server’s connect are retried, but only for failures in the connect phase, where nothing can have reached the site. A request that may have arrived is never retried, because a run claim is single-use. And one dropped ERP call now fails that call, not the whole run.

We also stopped a library from fetching a model price list from GitHub at start-up. On October 19 that fetch failed in 16 of 23 runs, and cost about 20 seconds each time.

What we got wrong

The 120-second window was measured from process start: the process’s age, read from /proc/self/stat against the boot clock. Our own diagnosis had recommended that anchor. On Cloud Run, that age over-reports, sometimes by more than the execution has existed. Three runs on the new runner, the same afternoon:

  • one run’s gate got about 11 seconds, not 120, before it gave up, because the clock said the window had already passed;
  • one logged “egress ready after 242.0 s” for an execution about 154 seconds old;
  • one logged “835.0 s” for an execution about 180 seconds old.

There was a second problem with the anchor. Before the gate, the runner fetches its run spec from our control plane, which on one run took 140 seconds. A window counted from process start spends that time too.

Fix two: start the clock where the wait begins

deadline = startup_clock() + STARTUP_WINDOW_S

The deadline is now set when the egress wait begins, and the same deadline bounds the claim and connect retries. The gate is the first thing that goes through NAT; anything before it talks to Google and shouldn’t use up the window. The process-age code is gone, and the logs now report the wait itself.

The tests run on a clock we turn by hand, starting at 950,000 seconds, a process up for eleven days. Under the old code that would have closed the window before it opened. As a mutation check, we anchored the deadline 242 seconds early, as the old code effectively did on one run; three tests fail.

The result

On the fixed runner, 3 of 3 runs succeeded. They waited 35.6, 27.5 and 3.5 seconds for the network, and one tool-server connect needed a retry. Three runs is a small sample, and we’re watching the logged waits. If they turn out too long, the other option is a different egress path for the runner.

What we’d tell someone else

And when you set a timeout, anchor it to the thing you’re waiting for. A process’s age is a different clock, and on a managed platform it can be wrong.