Skip to main content

DEPLOY.PREFLIGHT.REPLICATION_LAG — How far behind the replicas are, one moment before a migration adds to their work

  • Category: safety
  • Severity: info
  • Level: 0
  • Downtime class: online — the finding this check emits carries it. Lag costs a replica its freshness and blocks no writer on the primary
  • Stability: stable
  • Suites: deploy
  • Applies to: PostgreSQL and MySQL — every other driver, SQLite included, is never asked

A schema change reaches a replica whenever it catches up. Until then, an application reading from replicas is running against the old schema — and the failures that produces during the window, a column that does not exist yet, look like application bugs rather than like a deploy still in progress. On a large table that window is minutes.

This check asks how much of that debt already exists, immediately before the migration adds to it.

What it reports​

It reads live replication state through the run's activity reader, asking for it explicitly — the reader leaves replication out by default, and this check is the caller that turns it on, at the cost of one round trip.

EngineRead fromWhat the reading carries
PostgreSQLpg_stat_replicationone row per replica the primary streams to, its state, a time lag from replay_lag, and a byte distance from pg_current_wal_lsn() - replay_lsn
MySQLperformance_schema.replication_connection_statusone row per channel this instance replicates from, with its SERVICE_STATE — ON, OFF or CONNECTING, and no lag figure at all

Each replica in that reading is judged on its own, and produces at most one finding:

What the reading says about a replicaWhat the check does
its state is not streamingfail — one finding at high, naming the replica and the state word the server used
it is streaming and further behind than the linefail — one finding at high, naming both durations
it is streaming and at or inside the linenothing
it is streaming and the server reports no lag for itnothing — a null lag is not a zero one
there are no replication rows at allpass, carrying one info finding: no_replication_configured

Both arms are written in PostgreSQL's vocabulary, and the state test compares against the literal word streaming. MySQL's SERVICE_STATE never uses it, which decides what a MySQL run reports — see on MySQL the state test never matches below before reading this table as a description of a MySQL deploy.

The line is 10,000 ms, the shipped default of sqlens.preflight.replication_lag_ms, and the comparison is <=: a replica exactly 10,000 ms behind passes, one at 10,001 ms is reported. The message states both numbers with one decimal — "is 45.0 s behind, past the 10.0 s this run judged against" — because a duration without the scale it was measured against is not a reason.

Findings arrive sorted by replica name, so two runs over the same state produce the same report.

The disconnected replica is the sharper half​

Zero lag and no connection report the same number. One means caught up, the other means the replica is not receiving anything at all — and the second is the more dangerous, because a deploy that "looked fine on the replicas" was measured against a replica that stopped listening.

So the check never infers the connection from the lag. It asks the state directly, and a replica counts as receiving changes only when the server's own word for its state is exactly streaming — PostgreSQL's word, and only PostgreSQL's. Anything else produces the finding, whatever the lag column says:

The replica reporting-1 is not streaming — the server reports it as catchup. It is not behind by a little; it is not receiving changes at all, and a lag of zero would say the same thing.

No replicas is a pass that says so​

An instance nobody streams from is entirely legitimate, so it passes — but it passes with a reason. Left bare, that result would be byte-identical to "the replicas exist and are caught up", and those are two different sentences about a deploy. The info finding is how a report keeps them apart; it is not something the deploy did wrong.

Why it matters for a deploy​

The lag is not the damage. What it costs is the truth of what a replica serves, for as long as it lasts, starting from a moment you chose.

A migration lands on top of whatever the replica already owes. During the catch-up window, code reading through that replica sees the previous schema while the primary has the new one — so a read path fails on a column the primary already has, on an instance that was healthy a second ago, and the error surfaces as an application bug rather than as a deploy in flight. A replica that was already behind before you started widens exactly that window.

What to do about it​

The finding is a decision, not a defect, and there are only three answers:

  1. Wait. The replica is catching up and the lag is unusual here — let it drain and run the deploy afterward. This is the right answer whenever the number surprises you.
  2. Fix the replica. For the not-streaming finding this is the only answer. Nothing is catching up, so nothing will drain; the deploy would reach that replica whenever it reconnects, which may be after somebody has already read the old schema through it.
  3. Move the line. The lag is normal for this instance — a reporting replica routinely half a minute behind is a real configuration, not a fault. Say so, and keep the check for the cases it was built for:
// config/sqlens.php
'preflight' => [
    // Ten seconds is the shipped answer to "far enough behind that adding to it is a decision".
    // A positive integer, in milliseconds; a run falls back to the shipped default if it is not one.
    'replication_lag_ms' => 60000,
],

The finding names that setting itself, so the way out is in the message rather than in somebody's memory: "Move the line with sqlens.preflight.replication_lag_ms if this is normal here." Before this setting existed, a project whose replicas are legitimately seconds behind had one lever — switching the check off — and that takes the genuine cases with it.

Why the catalog says info and a finding arrives at high​

The two numbers answer different questions, and both are deliberate.

The catalog severity is info because what lag is acceptable depends on what the application does with its replicas, and the package cannot know that. The findings are emitted at high: by the time one exists, this run has a measured replica past a line this project set, and that is a concrete statement rather than a general one.

high rather than critical, in both arms and for opposite reasons. A lagging replica is a stale read, not a broken write, and the primary this deploy targets is unaffected. A replica that is not connected at all gets the same rating because there, nothing is catching up.

Why the findings are marked online​

Every finding this check produces carries online on the downtime axis, including the pass-side one. The lag costs nobody a lock and delays no statement — what it costs is the truth of what a replica serves. Calling that blocking would teach a reader to discount the field everywhere else.

When it cannot answer​

Two states are reported undetermined with a named reason, never as a pass:

  • The run has no activity reader, so how far the replicas are behind is unknown. The reason says the sentence that matters: "That is not the same as them being caught up — a deploy is about to add to whatever they already owe."
  • The read failed — a role without the privilege the view needs, for instance. The reason carries the server's own error message.

Both are prefixed replication_views_unreadable, and both block the run: sqlens:predeploy is fail-closed. A run blocked only by checks that could not answer proceeds with --allow-undetermined, which is the deliberate way out rather than a quiet one. See understanding undetermined and the deploy readers for the privileges each view needs.

What it does not claim​

  • It sees replication from one side only, and the side differs per engine. PostgreSQL answers from the primary's view of its replicas: a replica the primary does not know about is invisible here. MySQL answers from the replica's view of its source, so an empty answer there is a statement about this instance replicating from nothing. Either way, no_replication_configured is a statement about this instance, not a clean bill of health for a cluster.
  • On PostgreSQL a replica is named by its application_name, and one without a name is dropped. That is the name every finding here uses, so a WAL sender that set none is left out of the reading entirely — and if it is the only one, the check reports the no-replicas pass over a primary that is in fact streaming to somebody.
  • An empty list is not proof of a primary. The check reads the reading's replication list and nothing else. A PostgreSQL connection that reached a standby has no view of the replicas a primary streams to; the reader records that as a named skip in the reading, but this check's verdict is the no-replicas pass — which is why that pass says out loud that it describes this instance only.
  • It judges the time axis, never the byte one. PostgreSQL reports both and both travel in the reading, but only the millisecond figure is compared against the line. They disagree on purpose: a quiet primary keeps replay_lag small however much WAL is still outstanding.
  • On MySQL there is no lag figure to judge, and the state test never matches. The reading carries a channel and its SERVICE_STATE and nothing else, so the millisecond comparison never applies there. What remains is the connection-state arm — and it compares against PostgreSQL's word streaming, while MySQL answers ON, OFF or CONNECTING. A healthy MySQL channel is therefore reported as not streaming, at high, naming the word the server used (ON). Read a MySQL finding from this check as "this instance replicates from a channel", not as a verdict on that channel; the useful MySQL answer today is the no_replication_configured pass, which does mean what it says.
  • A lag the server does not report is silent. Reporting it as caught up would be a silent green; reporting it as behind would invent a number. The connection state is still judged.
  • It is a reading, not a watch. The state is the one that existed when the check ran, and it runs late in the preflight on purpose — a lag measured before the slower questions would already be from a moment that has passed by the time the report is written. A replica can fall behind a second later.

Sources​