Back to Field Notes

We put our marketing copy under test

Parts of this site described things the product didn't do. On October 19 we fixed every claim on a written list, then added a CI gate that fails the build if any of them come back. Run against the site as it was, the gate found 67.

Covers
Written
Repositories
infinary.io
Reading time
3 min
#marketing-site#ci#process
Contents

How the site got ahead of the product

This site was written ahead of the product, and some pages described what we meant to build as if it already existed. We had cleaned it up before, removing invented numbers in July and tying the agent catalog to agent definitions that actually run. But claims kept creeping back in, a sentence at a time, with every new page.

In October, a plan for the next round of marketing work began with an audit. It listed each claim on the site that the product didn’t back, with the file and line where it appeared and what was actually true. On October 19 we fixed the whole list in one pass:

  • How AI work is paid for. Some pages gave every AI action a small price of its own. Nothing is charged that way.
  • Who does the work. Some pages had the assistant finishing tasks itself. In fact, Aria prepares a change, and the person confirms it on their own login.
  • Spending ceilings. Some pages promised a hard ceiling on AI spending. For Aria, the budget only sends an alert; the hard cap applies to agent runs.
  • Data claims we couldn’t back. We removed them, and replaced them with one sentence that describes what happens: Aria runs inside your site and calls Google’s Vertex AI from your own Google Cloud project.
  • The chat on this site. It had been named after Aria, and described itself as knowing more than it can. It’s now Ask Infinary, and it says plainly that it can’t see your ERP. Aria is only ever the assistant inside your own system.
  • Kremio. Store badges and download buttons for an app that isn’t in any store yet. Kremio is an Android beta for Infinary customers.

The gate

npm run honesty:check scans every file under src/ and public/, sentence by sentence. Each rule is a pattern for one kind of false claim, sometimes with a second pattern that must appear in the same sentence. For example, a “sends” or “submits” only counts when the sentence is about the assistant.

Two details make it usable rather than noisy.

Negation. For most rules, a negation earlier in the same sentence excuses the match. “Nothing is billed per action” is the true statement, and it has to pass:

const NEGATION = /\b(?:no|not|nothing|never|without|cannot|until)\b|n't\b/i;

Some rules take no excuse, because the words themselves are the problem. The store rules are tied to the site’s own data instead: the day a store listing exists and its URL is filled in, that rule turns itself off.

A self-test. Before it scans anything, the gate runs its rules against the original wording of every audited claim, which must fail, and against the replacement wording, which must pass. If someone loosens a rule until it stops catching the claim it was written for, the gate fails on that, before it reads a single page.

Run against the site as it stood before the fix, the gate reported 67 failures, covering every row of the audit. On the fixed branch it reported none. It runs in CI with the catalog checks, and it has grown since: 17 rules and 41 self-test cases today, as pricing decisions added claims we may not make.

What it can’t do

A pattern can’t tell whether a new sentence is true. It can only keep known-false sentences from coming back. New claims still need a person who knows what shipped, and the gate gives that person a short list of things they no longer have to catch by eye.

These field notes go through the same gate. Writing them, we had to describe the false claims without repeating them word for word, or the build would have failed.

What we’d tell someone else

Treat marketing claims like product behaviour. When you find one that’s wrong, fix it, then write the wrong version down as a failing test so it can’t quietly return.