Size it before you build it.

Section 01 · the problem

Most export systems have one speed, and it is the slow one.

Asking for your data is a small request with an unbounded answer. A contact list might be forty rows or four hundred thousand, and the same button produces both. The safe engineering response is to send everything to a background worker, and that is what most systems do.

It is safe and it is wrong most of the time. The median export on this platform is one dataset of a few hundred rows, which takes tens of milliseconds to produce. Routing it through a queue, an object store, a signed URL and an email means the operator waits on their inbox for a file that was ready before the page finished reloading, and which they are sitting in front of.

The opposite failure is worse. Serialize everything in the request and a large workspace gets a timeout, or a process holding sixteen megabytes of CSV in memory per concurrent request.

So the interesting question is not how to build the file. It is how to know how big the file will be before building it, cheaply enough that the check is free, and reliably enough to act on.

Section 02 · the decision

An indexed count and a number per dataset.

Every dataset declares two things: a count query and a typical serialized row width. The estimate is their product. That is the whole mechanism, and it costs one indexed count.

The widths differ by more than an order of magnitude because the rows do. An agent row carries a composed brief that routinely runs to several kilobytes; a booking is a reference, two dates and two amounts. Treating them as interchangeable would make the byte ceiling meaningless for whichever one you guessed wrong about.

// Both bounds have to hold. Rows alone would wave through a few hundred
// rows of long descriptions; bytes alone would wave through a hundred
// thousand tiny ones.
func withinSyncBudget(rows, avgRowBytes int64) bool {
    if avgRowBytes <= 0 {
        avgRowBytes = defaultAvgRowBytes  // never treat unmeasured as free
    }
    return rows <= syncMaxRows && rows*avgRowBytes <= syncMaxBytes
}

flowchart TB
  REQ["export request"] --> N["normalize
  datasets + format"]
  N --> ONE{"exactly one
  dataset, as CSV?"}
  ONE -- no --> ASYNC
  ONE -- yes --> EST["estimate
  count x row width"]
  EST --> FIT{"under BOTH
  5,000 rows and 5 MiB?"}
  FIT -- no --> ASYNC["background worker
  build, store, sign, email"]
  FIT -- yes --> SYNC["serialize in the request
  return text/csv"]:::good
  SYNC --> AUD["audit row: ready, sized,
  already expired"]
  ASYNC --> AUD2["audit row: ready,
  object key + expiry"]
  classDef good fill:#DCFCE7,stroke:#16A34A

Both paths write an audit row. An export that skipped the worker is no less an export of customer data. The instant one is recorded ready, sized, and already expired, because there is no stored file to describe and a row implying otherwise is a lie the retention sweep would later trip over.

Section 03 · numbers

Four production catalogs, same query.

Server-side serialization of the real export query, jsonb aggregations for media and pricing tiers included, against four live workspaces spanning two orders of magnitude.

The shape is close to linear in rows and dominated by row width rather than row count. Sundora is a hundred times the rows of Khaas Food and nineteen times the wall clock, because its descriptions are a quarter the size. Three of the four finish inside fifty milliseconds, which is the case that would otherwise have gone to a queue and an inbox.

Only one of these needs the worker, and it needs it decisively: 14,881 rows and 16.4 MiB against ceilings of 5,000 and 5 MiB. There is no ambiguity at the boundary in real data, which is the useful property of picking two ceilings rather than one.

Section 04 · the honest part

The declared width is wrong by up to three and a half times.

Catalog rows are declared at 3,072 bytes. Sundora measures 1,155. Khaas Food measures 4,081. The same dataset, the same query, the same code, off by 3.5x in both directions depending on whose workspace it is.

The cause is language and habit. Khaas Food writes long Bengali product descriptions, and Bengali is three bytes per character in UTF-8 where English is one. Sundora imports terse supplier titles. A single global constant cannot describe both, and no amount of re-measuring fixes that, because it is a per-tenant property being estimated by a per-dataset number.

It has not caused a wrong routing decision, and the reason is the asymmetry the design leans on. Guessing small means serializing something large inside a request. Guessing large means sending a small export the long way round, which costs a worker cycle and an email. So the constants are set high and every miss so far has landed on the same side of the ceiling as the truth.

Stated rather than hidden, because the next person to touch the threshold needs to know the input is a heuristic with a factor-of-three error bar, not a measurement. The correct fix is sampling the workspace's own average rather than trusting a platform-wide constant, and that is written down where they will find it.

Section 05 · comparison

Against a conventional file pipeline.

The comparison worth making is not throughput. Both designs serialize CSV at roughly the same speed, because both are bounded by the database. The difference is everything wrapped around the serialization.

For the median request the practical difference is 28 milliseconds against a queue hop, an object write, a signature, an email send and however long the operator takes to switch to their inbox and back. The file is identical. The wait is not.

The trade is real and it is deliberate. The fast path holds the whole CSV in memory for the duration of one request, which is why it is bounded at 5 MiB rather than left to grow. Above that ceiling the worker is not a fallback, it is the correct tool, and the engine's job is to tell them apart before either has done any work.

Section 06 · after the download

A finished export is not a finished job.

An export is a complete copy of a customer list living in object storage behind a signed URL. The entire point of a short-lived signature is that it stops working, and for a long time nothing enforced that: the expiry was recorded and no job ever acted on it.

sequenceDiagram
  participant S as hourly sweep
  participant O as object store
  participant D as database
  Note over S,D: delete first, then flip
  S->>O: delete object
  O-->>S: ok (missing key counts as ok)
  S->>D: status expired, clear link and key
  Note over S,D: crash between the two? the row is
still ready, so the next sweep retries

// Deletes are idempotent, so a sweep that dies between the two steps
// retries next hour and finishes the bookkeeping then. The other order
// marks a row expired and leaves the file behind, and nothing ever looks
// at it again to notice.
if err := store.Delete(ctx, key); err != nil {
    return reapFailed   // leave the row ready: that is what brings us back
}
return repo.ExpireExport(ctx, row.ID)

The intuitive order is the dangerous one. Mark the row expired and then delete: if the delete fails, no future sweep selects that row, and the file is stranded permanently with nothing left in the system aware it exists. Deleting first fails toward doing the work again, which is the only acceptable direction when the work is destroying customer data.

Two behaviours hang off the same sweep. Re-signing a lapsed link on read is bounded by the existing retention deadline and never extends it, or reading an export would quietly keep it alive. And erasing a workspace retires its exports immediately rather than waiting for their own expiry, because a copy of exactly the data an erasure destroys is not something to leave lying around for another six days.

Section 07 · inbound

The other direction, and why it is a different shape.

Import runs through the file extraction pipeline rather than this engine, and the asymmetry is worth naming. On the way out the platform knows exactly what it is producing. On the way in it knows nothing and has to be defensive about all of it.

Spreadsheets arrive as CSV or XLSX and are normalized to text by a single extractor before anything else touches them, so the rest of the platform never learns which of the two it was. Content is truncated on a UTF-8 boundary rather than a byte offset, which sounds pedantic until a Bengali catalog is cut mid-character and the whole row becomes unparseable.

An export is bounded by a number you can compute. An import is bounded by whatever someone uploaded, which is why the two paths do not share a size model: one estimates, the other validates.