- scheduled jobs now run at admission
- 31
- checks passed / failed with Redis 8
- 128 / 0
- with Redis 6.2: the bug, caught
- 126 / 2
- unreachable during the local swap
- 21 s
What we saw
On October 20 we installed Frappe Helpdesk 1.31.0 on erp.infinary.io, our own site. On every scheduler tick, Helpdesk runs helpdesk.search.build_index_if_not_exists, which builds its knowledge-base search index in the site’s Redis cache.
It failed on every tick, and almost nothing said so. There was one row in the Error Log, from the index build that the install queues once, and nothing per tick. The background-job queue reported each run as “Successfully completed”, and the scheduler log was empty. We stopped the job by hand.
Why nothing was logged
Frappe runs scheduled jobs through ScheduledJobType.execute, which catches the job’s exception. For jobs that run every tick, and for cron jobs, it also writes no Scheduled Job Log at all, because create_log is 0 for those types. So a job that raises on every tick leaves no record of failing. Jobs queued in the background are different: they log through the normal job path, which is where our one Error Log row came from.
Why it failed
The index build uses FT.CREATE, from Redis’s query engine. Our servers ran the Redis image from Frappe’s upstream compose file, redis:6.2-alpine, which has no FT.* commands at all.
What breaks is narrower than it first looked. Only Helpdesk’s knowledge-base search needs Redis: article search in the reply editor, the articles suggested on a new ticket, and one report. Ticket search uses Frappe 16’s SQLite search and works fine on Redis 6.2. Helpdesk has no setting that moves knowledge-base search to SQLite, though; the only way around Redis would be to override Helpdesk’s search with our own.
What we compared
We replayed Helpdesk’s 15 search calls, plus Frappe’s own cache checks, against each candidate:
| Image | Helpdesk’s 15 search calls | Image size (compressed) | Idle memory |
|---|---|---|---|
redis:6.2-alpine (what we ran) | none | 11 MB | 7 MB |
| Redis Open Source 8.10.2, Alpine | 15 of 15 | 39 MB | 23 MB |
| Redis Stack Server 7.4.0 | 15 of 15 | 298 MB | 23 MB |
| Valkey bundle 9.0.6 | none (no STOPWORDS, FT.ADD, SYNUPDATE or SPELLCHECK) | 55 MB | 45 MB |
We chose Redis 8.10.2. Redis Stack works too, but its image is eight times the size and hadn’t been updated in almost a year.
What we changed
Fix 1: Redis 8 for the cache only. Our bring-up script now pins Redis 8.10.2 by digest for the site’s cache, and leaves the queue on 6.2. A cache-only mode moves an existing server across. Rehearsing it locally found two things a runbook would have missed:
- Frappe has to restart. Each Frappe process turns its client-side cache off for good when its Redis restarts. Frappe’s own comment says “there’s no easy way to recover from this”.
- The web server has to restart last. The first rehearsal restarted the backend but not nginx, which had resolved the backend’s address once, at start. The restarted containers came back on new addresses, so nginx answered 502 on every page.
With both restarts, the swap took 27 seconds locally, and the site was unreachable for 21 of them. After it, the index built, knowledge-base search found its articles, and an invoice, an employee and a CRM lead all saved normally, with no new errors over five more ticks.
Fix 2: admission now runs scheduled jobs. Before an optional app is offered, a rehearsal installs and removes it on a copy of our image. That rehearsal now also calls every scheduled job each app declares, once and directly, so an exception surfaces. Then it runs three real scheduler ticks with logging turned on, and fails on a raised exception, a new Error Log row, or a failed job log, naming the job.
With the Redis 8 cache, all 31 scheduled jobs of HRMS, CRM and Helpdesk passed: 128 checks, 0 failed. Pointed at the old Redis 6.2 cache, it failed exactly 2 checks, both naming build_index_if_not_exists and unknown command FT.CREATE. That is the bug we hit on our own site, caught before an install.
Where it stands
Both fixes are merged. As of October 20, our own site still ran Redis 6.2, with the Helpdesk job stopped, until we picked a quiet half-minute to move it. We moved it the next morning, October 21: the site’s cache is now Redis 8.10.2, and the Helpdesk job is running again with its search index in place. New servers get Redis 8 once the control plane that provisions them is redeployed.