Skip to main content

Outages on the record — what actually went wrong, and what this catches

Every rule here exists because somebody's database did something expensive. This page lists the outages that are on the public record — someone wrote them up, the write-up is still reachable, and the date it was read is recorded beside it.

It is short, and that is the point. A list of incidents is only worth reading if nothing on it was invented to fill a row, so an entry appears when a source exists and not otherwise.

What each entry claims, and what it does not​

Two separate questions, kept apart because collapsing them lets the weaker answer be read as the stronger one:

Prevented bythe change would not have been written that way — the rule refuses it before it runs
Detected bythe change still happens, and the state it leaves is found and named

"We would have found it afterwards" is a real thing to offer and a smaller thing than "it would not have happened".

A foreign key queued behind a long read​

PostgreSQL · engine · the source states late January and gives no year, so none is recorded here. Inferring one from the article's edit stamp would be inventing it.

Adding a foreign key takes an ACCESS EXCLUSIVE lock on both tables. A long-running read on the parent table was still holding its own lock, so the migration queued behind it — and a queued exclusive lock blocks everything that arrives after it, including queries that would not have conflicted with either. About fifteen seconds of unplanned API downtime, from a migration that touched no data.

Two rules, two independent stops. Adding the constraint NOT VALID and validating it in a second statement never takes the exclusive lock at all; a lock timeout would have made the migration fail fast instead of holding the queue open. Either alone is enough, which is why both are named rather than one chosen.

A concurrent index build that SQLens would not have prevented​

PostgreSQL · 2018-03-12 · roughly ninety minutes

CREATE INDEX CONCURRENTLY was used exactly as recommended — including by this package — and a defect in PostgreSQL 9.6.6 turned it into contention anyway: vacuum cleaning a GIN index's pending-insertions list collided with the build, inserts into the notes table blocked for fifteen seconds and then failed, and comments on issues and merge requests were unpostable. The defect was fixed in 9.6.7.

This entry is here because it is not a win. A migration written the way every guide says to write it still took a service down, and no rule on this list would have stopped it. What is on offer afterwards is narrow and worth saying plainly: a concurrent build that fails leaves an INVALID index behind, and that gets found and named rather than discovered months later by a query plan.

Read it as an argument for the post-deploy check, not against the rule — and as the reason to believe the entry above it.