Two levers for a growing brief.

Section 01 · the problem

A brief you re-send every turn is a bill you pay every turn.

The system brief that gives an agent its identity, rules, and worked examples is composed once and then sent to the model on every single turn. As a brief grows past a few thousand tokens, that is a recurring per-turn cost, a per-turn prefill latency, and a standing draw against the model's usable context window.

The obvious lever is the wrong one. We shipped a provably lossless whitespace compaction (Compact, in internal/agentbrief/compact.go) and measured what it saved on a real, mature brief: zero tokens. The composer already emits tight markdown, so there is no free whitespace to reclaim. The tokens that could shrink are the prose in Instructions and Samples, which are exactly the tokens where a reword changes behaviour. That honest result is what forces the real question: if the safe lever saves nothing, what actually moves the number without moving the agent?

Section 02 · two levers, not one

Caching cuts the cost. Compression cuts the size. They are not the same tool.

These are complementary, not competing. Caching makes the same prefix cheap to re-send; compression makes the prefix smaller. One is a runtime property with zero behavioural risk; the other is an authoring decision that has to be reviewed.

flowchart LR
  subgraph AUTHOR["① AUTHORING TIME · operator"]
    I["brief items
(agent_prompt_items)"] --> C["compose"]
    C --> K["Compact · lossless
whitespace only"]
    C -.->|"opt-in per prompt"| Z["Compress · model
reviewed diff"]
    K --> P["publish → agents.system_prompt"]
    Z -.->|"accept = Update"| P
  end
  subgraph TURN["② EVERY CUSTOMER TURN · runtime"]
    P --> PRE["stable system prefix"]
    PRE --> PROV["provider via CF AI Gateway /compat"]
    PROV -->|"cache hit?"| BR[("bot_responses
cached_input_tokens")]
  end

Section 03 · make it measurable first

You cannot tune what you cannot see. So we made cache hits visible.

The brief is the textbook case for provider prompt caching: a long, stable prefix re-sent constantly. On a cache hit, cached input is billed at roughly a tenth of the rate. But we had never confirmed whether cache hits actually flow through the path we route on, because the number was being thrown away.

Production routes every model through the Cloudflare AI Gateway /compat endpoint in OpenAI-compatible wire format. The usage frame on that path carries the cache-served portion of the input as prompt_tokens_details.cached_tokens (the Anthropic-native path reports the same thing as cache_read_input_tokens). The stream parser read prompt_tokens and completion_tokens and dropped the cached breakdown on the floor, so a cache hit was invisible.

The change is deliberately small and additive: parse the cached count already on the wire, thread it through the dispatcher, and persist it in a new ClickHouse column bot_responses.cached_input_tokens (migration 011, auto-applied on boot). It is a subset of input tokens, not additive. Nothing about what is sent upstream changes. The whole spike then reduces to a one-line query: read the column, and a non-zero value on a repeat turn is proof caching is flowing.

The principle: the first move on a performance question is not to optimise, it is to instrument. A cache lever with no cache metric is a guess. The cheapest, zero-risk change was to capture a number that was already being sent and then discarded.

Section 04 · compression, reviewed

The model may propose a shorter brief. It may never apply one.

Model-driven compression is the only lever that shrinks the prose caching cannot. It is also the dangerous one, so the entire flow is built around a single rule: nothing reaches the live agent without an explicit human accept.

flowchart TB
  REQ["POST /agent/brief/items/:id/compress"] --> RUN["Service.Compress
runs CompressSystem, writes nothing"]
  RUN --> G{"deterministic invariant guard
numbers · prices · urls · code fences"}
  G -->|"drops → flags"| DIFF["operator reviews the diff"]
  G -->|"clean"| DIFF
  DIFF -->|"Accept"| U["Update → auto-version
earlier body one Restore away"]
  DIFF -->|"Discard"| N["nothing written"]
  U --> LIVE["live agent"]

Section 05 · the deterministic guard

A human skimming a diff misses a dropped price. Code does not.

A reworded paragraph reads fine even when it quietly lost a number. So before the operator ever sees a proposal, a deterministic guard runs, with no model involved, and flags any fact the shorter version dropped.

The guard extracts every fenced code block, URL, price, and number from the original, then checks each survives in the compressed output. It works by membership, not by count: a value that appears at least once in the shorter version is not flagged, so ordinary rewording stays quiet and only a genuine drop is loud. Code, URLs, and prices are scanned first and their spans blanked before the loose-number pass, so a digit inside a surviving price is never double-reported. It is cheap, needs no LLM, and catches the single highest-risk silent change: a lost link or figure.

Where the line is drawn (decision D2): rule-type prompts are held out of model compression entirely in v1. A rule is a limit the agent must never cross, and it is where a reword is most dangerous, so a rule is whitespace-compacted only and its wording is left byte-for-byte. Every other type is eligible, and every proposal, even a guard-clean one, is still reviewed. There is no auto-accept, because auto-accept would reintroduce the exact silent-drift risk the feature exists to prevent.

Section 06 · why both, given the growth

Size hits three axes. Caching only fixes one of them.

It is tempting to pick one lever. The growth of the brief is exactly why you cannot.

Do caching first, because it lands the cost half of the problem with zero behavioural risk. Do compression for the two axes caching cannot touch: guaranteed latency and the context budget. The bigger the brief gets, the stronger the compression case becomes on exactly those two.

Section 07 · design principles

Two rules the whole design obeys.