One tenant took the whole connection pool

Engineering · 5 min read

Our tenant isolation was correct. Two tenants, separate realms, separate apps, RLS confirmed both ways — the victim's token returns [] for the storm tenant's table, and a script asserts it on every run. Nobody could read anybody else's rows.

Then we pointed a load storm at one of them and the other one fell over.

What we measured

Tenant A issued heavy 200-row reads at 60 concurrent. Tenant B issued the light 100-row read a grid does when you open it. Same stack, same database, 2,000 rows each.

Tenant B degraded 322× against a published bound of 2×, and 10.5% of its requests were refused outright. The 10,007ms p99 is a timeout ceiling, not a measurement — we never found out how slow it really was.

These are single-machine figures: Docker, the harness and the browser all on one box. That is not a production result and we are not going to present it as one. It is the right shape of measurement for the question — does one tenant's load reach another tenant — and the wrong shape for "how fast is Appdor".

The part that made it obvious this was contention and not a bug: the storm tenant's own throughput collapsed to 12.7 req/s. The instance did not throttle one tenant in favour of another. It fell over for both. And recovery was immediate — 5ms p99 the moment the storm stopped.

The mechanisms existed and nothing called them

This is the uncomfortable part. The fairness module was already in the tree. It had tests. The only function of it the running product ever reached was the one that publishes the promise to the capability catalogue.

A module that is imported by nothing on the request path is documentation with a test suite. We had shipped the paperwork.

The fix was in-flight caps, not rate limits

The controller that now sits in front of PostgREST caps concurrent in-flight work per tenant. That distinction is the whole result:

The "after" column is the 2026-09-07 re-run against the current gateway: 3,000 storm requests at 60 concurrent, two live tenants with tokens minted for separate users. Baseline p99 28ms, under storm 25ms, recovered 15ms.

The defensible claim is "well inside the bound", not a specific ratio, and that is deliberate. The same harness has returned 0.50×, 0.14×, 0.73× and 0.89× across runs of unchanged code, because the ratio is computed against whichever baseline the first phase happened to measure — and a cold stack measured 200ms p99 where a warm one measured 44ms. A sub-1.0 ratio is evidence the baseline is still warming, not evidence that a storm makes the victim faster. Quoting the prettiest of those numbers would be the same species of mistake as the one this post is about.

The storm's request rate was modest. Its concurrent count was the entire connection pool. A token bucket would not have fixed this, and we can show that rather than assert it: at the shipped 200/s limit the rate limiter absorbed 2,308 refusals; raising it to 5,000/s moved every one of those to the queue and changed the outcome not at all — 85% shed either way. When you can move a mechanism's threshold by 25× and the result does not care, that mechanism is not the one doing the work.

A wider queue made it worse

We tried a 200-deep per-tenant queue on the theory that shedding less is kinder. It converted refusals into waiting: the storm took 4.9 minutes instead of 7 seconds, and the victim was still at 2.41× baseline when the run ended. It failed the recovery check.

Refusing a request in 1ms is a better answer than serving it in 10 seconds. Queues are how you turn a load problem into a latency problem and then lose track of it.

What we still cannot claim

The criterion this was measured against is still marked unmet in our own audit, and that is the most useful sentence in this post.

It asks for a sustained automation event storm and a public API burst, simultaneously. The harness drives the burst only. So the measurement supports a narrower claim than the criterion makes, and the honest version is "we fixed the case we can drive" rather than "we meet the bar we set ourselves". The audit row says so in the same words; we would rather you read it from us.

The other thing the ratio cannot tell you is why. A failure here says the promise broke, not which mechanism broke it — that is what the per-tenant gauges are for. A fairness number is a smoke alarm, not a diagnosis.

Why this is in a spreadsheet-database you run yourself

Appdor runs as one container you start yourself, so "multi-tenant" is not an abstraction our SaaS operator worries about on your behalf — it is your Postgres, your pool, and your two departments. The isolation you get from RLS is about reading. Nothing in RLS stops one tenant from consuming the resource every other tenant needs, and the failure looks like a slow app, not like a security incident, which is why it survives so long.

The storm is a script in the repo:

npm run test:fairness

It fails loudly, in the direction of the number above, if the controller ever comes unwired again — which is the only reason we trust it now, having already been wrong once about a module that existed.