Five issuers or fifty, in the same three days

The shape of the problem

Statements go out at month end. That is true of banks, of insurers, of utilities, and it is true of all of them at once.

So the load on a document distribution platform is not a steady rate with a peak. It is a few days of everything, then three weeks of almost nothing. Five to fifty organisations uploading inside the same seventy-two hours, then silence until the next month.

Month-end (1st–5th):      5–50 issuers upload concurrently
                          500K–5M documents in a 2–3 day window

Rest of month (6th–30th): near-zero activity
                          occasional ad-hoc batches

Those figures are design targets, not measurements. We are describing what the system was built to absorb, not a throughput we have benchmarked in production, and the distinction matters more to us than it probably does to you.

The consequence is the interesting part. The challenge is not processing one large batch quickly. Plenty of things do that. It is handling many concurrent batches in the same window without one issuer’s month-end starving another’s, and then not paying for that capacity for the other twenty-five days.

What that ruled out

A single process working through a queue of documents is the obvious design, and it is the wrong one here. It makes every issuer wait behind whoever uploaded first.

So the unit of work is the batch, not the document. Each batch gets its own worker, and the cluster scales workers on queue depth rather than on a schedule. Month end brings the capacity up, and the sixth of the month takes it away again.

Isolation, argued out rather than assumed

Multi-tenant financial documents make tenant isolation a compliance requirement rather than a design preference. The options are the usual ones: a tenant column on shared tables, a schema per tenant, or a database per tenant.

We took schema per issuer. What is worth showing is not the decision but the argument, because it was not unanimous and the objection was a good one.

## Status

Accepted

### Team Review (2026-01-20)

**Concern raised:** Migration complexity — running migrations against all
schemas adds operational overhead. Alternative proposed: shared tables with
dedicated instances for customers needing high isolation.

**Resolution:** Team consensus reached in favour of schema-per-issuer based on:

1. **Migration automation** — single script via GitOps, zero manual
   intervention post-setup
2. **Bounded scale** — 50–100 issuers max, parallelizable, ~2 minutes total
3. **Archival simplicity** — pg_dump --schema + s3 sync vs multi-table DELETEs
   with FK ordering
4. **Security guarantee** — "can't access wrong tenant's data because it's
   just not there"

The concern was real: migrations against a hundred schemas is operational overhead that a tenant column does not have. It was answered with numbers rather than opinion — bounded at fifty to a hundred issuers, parallelisable, about two minutes — and with the observation that archival gets dramatically simpler when a tenant is a schema you can dump and sync rather than rows to delete in foreign-key order.

The line that settled it was the security one. You cannot query the wrong tenant’s data if it is not in the schema you are connected to. Every other model defends that boundary with application code, and application code has bugs.

Deliberately boring choices

Two of those are worth defending, because they look like omissions.

No web framework. Go’s standard library net/http does routing, middleware and handlers perfectly well. A framework would add a dependency, a migration path, and an abstraction between us and behaviour we need to reason about precisely under load.

No ORM. database/sql and written SQL. When tenant isolation is enforced at the schema level and correctness is a compliance question, the last thing we want is a layer generating queries we did not read.

Both are choices we would defend in review, which is the test. Neither is laziness, and both would be wrong on a different project.

Where it is

The platform is built and running. Three-phase pipeline, extract then match then distribute, over WhatsApp, SMS, email or CSV. Multi-stage builds to a distroless image, Kubernetes with Kustomize, Flux moving releases.

What it does not yet have is a production month-end at full scale behind it. When it does, that number will replace the design target above, and this page will say so.