From a silent call to 820 milliseconds.

Section 01 · the result first

The greeting was the easiest 2 seconds on the whole call.

Every call opens with the same sentence, in the same voice, in the same language. It was being synthesized from scratch every single time, at 2.2 to 2.7 seconds of vendor round trip, for audio we had already paid for on the previous call.

Getting there took a long stretch of work that was mostly not about latency. Before anything could be timed, calls had to stop being silent, the agent had to remember what the caller said one turn ago, and it had to stop discarding answers to its own questions. Then the profiling could start, and the profiling produced two results that pointed the opposite way from the intuition.

Section 02 · connected and silent

Four separate bugs produced exactly the same symptom.

A call that rings, answers, and then plays nothing is the least informative failure in this stack, because every stage can cause it and none of them says so. We hit four causes in a row, and each one was invisible until the one in front of it was fixed.

The line that made all four hard to see

Three call sites in the turn loop finished a synthesis with a discarded error. That reads as harmless, and on a streaming synthesizer it nearly is. On a request and response synthesizer it is the opposite: the terminal call is where the vendor request actually happens, so it is the only place a rejection can surface.

// Done() is where a request/response synthesizer actually calls the vendor, so
// this is where a TTS failure surfaces. Discarding it meant a rejected
// synthesis produced zero audio, no error and no log: the runner answered
// 200, the turn "succeeded", and the caller heard nothing.
var ttsErr error
if opened {
    ttsErr = sink.tts.Done()
    <-drained // all audio enqueued
}

With the error dropped, a rejected synthesis produced zero audio, no error, and a turn that reported success. The turn counter went up. The latency metric recorded a healthy number. The caller heard nothing. Any failure path that increments a success metric will outlive several rounds of debugging, because it removes itself from the list of suspects.

The general rule we took from this: in a media pipeline, treat "produced no output" as an error condition with its own alarm, separately from "returned an error". The two are the same event to a caller and completely different events to the code.

Section 03 · state

Every caller utterance was turn one of a brand new conversation.

The endpoint that runs a turn minted a fresh session per request. Each utterance therefore arrived at the language model with no history, as the opening line of a conversation that had never happened.

This did not arrive as one bug report. It arrived as three, from three different people, that read like three unrelated quality problems:

• It greets me every time. Correct behaviour for turn one, performed on turn four.
• It forgot my name immediately after I gave it. The name was in a transcript the next request never saw.
• It lost the booking between my name and my phone number. Two consecutive turns, two unrelated conversations.

One cause. The fix is that the bridge, which is the only component that lives for the length of the call, keeps the transcript and sends it with every turn. That is the right place for it: the turn endpoint is stateless by design and the media socket is the natural lifetime of a call.

Worth naming, because it recurs: three convincing, specific, independently reported symptoms can share one line of cause. Fixing them as three tickets would have produced three workarounds and left the session bug in place.

Section 04 · turn-taking

The agent asked a question and threw the answer away.

A final transcript that landed while the agent was still speaking failed a state-machine guard, and the utterance was discarded with a single line in the log. On a phone call that is the caller answering a direct question and being ignored.

sequenceDiagram
  participant C as Caller
  participant S as Session state
  participant A as Agent
  Note over C,A: before · the guard wins
  C->>S: final transcript arrives while agent is Speaking
  S-->>A: BeginTurn refused
  A->>A: utterance dropped, one line in the log
  Note over C,A: after · the caller wins
  C->>S: final transcript arrives while agent is Speaking
  A->>S: bargeIn · Speaking becomes Listening
  A->>S: BeginTurn accepted
  A->>C: answers what the caller actually said

Overlap is normal on a phone call, not an error. The detector at the telephony edge usually clears playback before this happens, but it only fires while the platform reports the agent stream as playing, and the pacer on the bridge can still be draining. There is a window on either side of every media start and stop event where the two disagree about who is talking.

The corrected order is: cut the audio, move the session out of speaking, then begin the turn. One state that genuinely cannot overlap with itself is thinking, because final transcripts are handled serially, but a greeting or an idle prompt synthesizes from its own goroutine. A short bounded wait covers that race instead of dropping what the caller said.

Section 05 · the one we caused

A nil pointer, a full green test suite, and every call panicking twice.

The clock is injected so that timing can be tested without sleeping. It was defaulted inside a constructor, and the constructor took its configuration struct by value.

// withDefaults fills in what an injected Deps may legitimately leave unset.
//
// It exists because the clock used to be defaulted inside newAgent, on
// newAgent's own by-value COPY of Deps. That left the caller's copy with a
// nil clock, and RunAgent, which also calls deps.clock(), dereferenced nil
// and took the whole call down with it. Normalize once, at every entry
// point, before anything reads a field.
func (d Deps) withDefaults() Deps {
    if d.clock == nil {
        d.clock = time.Now
    }
    return d
}

The constructor filled in its own private copy. The caller kept the copy with the nil field, and the request handler, which calls the clock directly before the constructor ever runs, dereferenced it. Every call panicked twice, the socket died, and the telephony scenario did the only sensible thing it could: it fell back to the recorded apology about a technical problem and asked the caller to try later.

The interesting part is why the tests passed

Coverage on this package was good. Every unit test constructed the inner object directly and injected a fake clock, which is exactly what you want for testing a pacer against fixed timestamps. Not one of them went through the real request handler, because the real handler takes a concrete WebSocket type and standing one up in a test is more work than the test being written at the time was worth.

So the tests proved the object behaved correctly when constructed the way tests construct it. The panic lived entirely in the gap between that and the way production constructs it, which no test occupied.

If a component is only ever built by its tests one way and by production another way, the tests are describing a different program. That is worth a targeted test at the seam, and it does not need to be a full end-to-end suite to be worth having.

Section 06 · endpointing

The defaults chopped one Bengali sentence into three.

Reported as "the voice processing is jittery". The recording showed one continuous utterance registering as three: speech start at 101.4, end at 103.8, start again at 104.3, end at 105.8.

Every fragment ends a turn. So the agent answered the first third of a sentence, started speaking, and then barged in on itself when the second third arrived. From the caller side that is not a tuning problem, it is an agent that cannot hold a thought.

// Tuned against a real bilingual call where the defaults chopped one
// utterance into three:
//   speechStart 101.4 / end 103.8 / start 104.3 / end 105.8
// Every fragment ends a turn, so the agent answers half a sentence and then
// barges in on itself. That is most of what "the voice processing is
// jittery" actually was.
vad: { threshold: 0.6, minSilenceDurationMs: 700, speechPadMs: 150 },
turnDetector: { threshold: 0.5 },
bargeInGuardMs: 400,

The language matters here and the defaults do not know it. Bengali carries longer pauses inside a sentence than English does, and any caller reading out digits pauses between groups regardless of language. A 300 millisecond silence window ends the turn in the middle of a phone number, which is precisely the moment in a booking flow where you can least afford it.

These four values are per-deployment configuration, not constants. A tuning that is correct for one language and one line quality is a guess for the next one, so they live in the call scenario config with the reasoning written next to them.

Section 07 · geography

Identical build, two calls minutes apart, opposite outcomes.

Same binary, same configuration, same agent, same phone number. The only difference was the network the caller dialled from, and it decided which region the telephony platform allocated a media server in.

Call B routed audio from a Singapore media server to a bridge in us-east and back. The handshake alone took 3.1 seconds and then 8.3 seconds on the retry. Not one speech event was ever detected, the agent managed under a second of audio in total, and the caller gave up at 15 seconds. There was no bug to find, and we spent real time looking for one.

Check the media server region before you touch the code. A voice bug that reproduces on one network and not another is a topology question until proven otherwise. This is also the argument for co-locating the bridge with the telephony point of presence: it is the only part of the budget that is pure geometry.

Section 08 · the measurement

Synthesis is 59 percent of the wait, and it is linear in characters.

Once calls were reliable, five representative questions were run against the live stack and each stage timed. Mean language model time was 1.45 s. Mean synthesis time was 2.07 s. Synthesis is the larger half of everything a caller waits through.

flowchart LR
  C["Caller stops speaking"] --> V["Turn detector
700ms of silence"]
  V --> S["Speech to text
final transcript"]
  S --> L["Language model
mean 1.45s"]
  L --> T["Speech synthesis
mean 2.07s"]
  T --> P["Pacer
one 20ms frame per 20ms"]
  P --> A["Caller hears the reply"]

The useful answers are the slow ones

Synthesis time on this vendor fits a straight line: 0.24 seconds of fixed cost plus 12.2 milliseconds per character. That fit reproduces all five measurements to the nearest hundredth of a second, so it is a good enough model to plan against.

The consequence is uncomfortable. A 19 character reply costs 0.47 s of synthesis; a 317 character reply, which is the one that actually answers the question about what to bring, costs 4.11 s. The most useful answers are systematically the slowest, and the slowness is proportional to how useful they are. Any optimisation that works by shortening replies is trading the answer for the timing.

Section 09 · negative result one

Changing the language model buys 0.6 seconds, mostly by saying less.

The instinct when a turn feels slow is to reach for a faster model. Five were run against the same five questions on the same stack.

A 0.6 second improvement is real, and it is not where the problem is. More to the point, the spread across these five is not a spread in raw generation speed. It tracks how long each model chose to make its replies, and reply length is the input to the stage that dominates the budget. The model choice is mostly a proxy for verbosity.

This reframes the question. Instead of "which model is fastest", the question is "what determines reply length, and can we control it without controlling the answer". Which leads directly to the next experiment, and to the wrong answer.

Section 10 · negative result two

We told the model to be brief and it got slower.

If synthesis is linear in characters, capping characters caps synthesis. So the agent brief gained an explicit instruction: at most two sentences, at most 200 characters. Then the same five questions were run again.

The short answers stayed short, in the 69 to 84 character band they were already in, because they were never the problem. The two list-shaped answers, the ones the cap was written for, both got longer. Mean synthesis time went up from 2.07 s to 2.34 s. The change was reverted.

We did not chase the mechanism far enough to be certain of it, and the honest summary is the measurement: an instruction of this kind is not a reliable control surface for output length on a list-shaped question. If reply length has to be bounded, it needs to be bounded by something deterministic outside the model, or the answer needs to be restructured so brevity is natural rather than requested.

Recording this mattered more than the 0.27 s it cost. Left unmeasured, "we tell it to be concise" would have stayed in the brief as a latency control that everyone believed in and nothing supported.

Section 11 · the win

Some of the audio is byte-identical on every call, so pay for it once.

The greeting, the acknowledgements, the idle nudges, the farewell, the line that asks the caller to repeat themselves. Every one is a fixed string in a fixed voice, and every one was being re-synthesized on every call.

flowchart TB
  T["Synthesize some text"] --> D{"marked cacheable?"}
  D -->|"no · a generated reply"| V["vendor round trip"]
  D -->|"yes · a fixed phrase"| K["key = text + language + voice"]
  K --> H{"already rendered?"}
  H -->|"hit"| OUT["audio, no network"]
  H -->|"miss"| V
  V --> ST["store, if cacheable"]
  ST --> OUT
  BOOT["boot · background warm"] -.->|"pre-render 21 phrases"| K

Caching everything is the trap

The obvious version of this feature caches all synthesis output. It is worse than useless. Generated replies are never byte-identical twice, so an everything-cache pays full storage for a hit rate near zero, and then evicts the handful of fixed phrases that would have hit. So caching is opt-in per phrase: the caller marks a request cacheable, and nothing else is stored.

That also removes the need for an eviction policy. The cacheable set is a small fixed list known at configuration time, so a least-recently-used policy could only ever discard something about to be needed again. There is a hard entry cap as a guard against someone marking something unbounded as cacheable, and past it synthesis still works, it just stops being stored.

Instant acknowledgements

The second half is cheaper and does more for how the call feels. A turn costs about 1.5 s of model plus 2 s of synthesis before the caller hears a syllable, and up to 6 s on the long answers. That silence is what makes a caller repeat themselves or start talking over the reply.

sequenceDiagram
  participant C as Caller
  participant B as Bridge
  participant L as Language model
  participant T as Synthesis
  C->>B: stops speaking
  B->>C: acknowledgement from cache, about 1s
  par behind the acknowledgement
    B->>L: run the turn
    L-->>B: reply text
    B->>T: synthesize the reply
    T-->>B: audio
  end
  B->>C: reply plays straight on the end of it

This does not make the answer arrive sooner. It makes the caller stop waiting in silence, which is the part they experience. A human receptionist does exactly this. Two properties are load-bearing: the phrases rotate, because the same acknowledgement before every answer is a sharper robot tell than the silence it replaced, and they stay short, because the acknowledgement plays before the reply so its length is added to the total.

It is queued straight into the pacer rather than going through the normal speak path, because that path opens its own turn and the acknowledgement has to live inside the turn that is already running. Failures are swallowed on purpose: comfort noise must never fail the turn that carries the actual answer.

Section 12 · what is next

Streaming synthesis, because it helps every turn.

The synthesis driver currently uses a request and response endpoint. It sends the whole reply, waits, and gets the whole audio back. A 363 character reply therefore waits the full 4.7 seconds before one byte of it plays.

The same vendor offers a streaming WebSocket. Audio would start after the first sentence rather than after the last one. On the linear fit that is roughly 3.5 seconds off the long answers and 1.5 to 2 seconds off the average. Unlike the cache, which only helps phrases that repeat, this helps every generated reply, which is every turn that matters.