mariadb-errors-replication-lag
Installation
SKILL.md
MariaDB Replication Lag
How to measure, diagnose, and fix replication lag on a MariaDB replica. Covers the unreliability of Seconds_Behind_Master, parallel-applier tuning, long-master-transaction root causes, GTID position checks, and the semi-sync-to-async fallback that looks like a lag spike but is not.
Quick Reference
Seconds_Behind_MasterfromSHOW SLAVE STATUSis UNRELIABLE for real-time lag : it updates only between binlog events and reports0while the applier is mid-replay of a large transaction. NEVER use it alone for alerting. ALWAYS pair withpt-heartbeat(Percona Toolkit) for true lag.slave_parallel_threadsdefault is0(single-threaded applier). On any modern workload, set to4-16based on disk-IO benchmark. The value is dynamic but requiresSTOP SLAVE ; SET GLOBAL ... ; START SLAVE.slave_parallel_modedefault isoptimisticfrom 10.5.1+ (wasconservativeuntil 10.5.0). NEVER set tononein production unless you are reproducing a determinism bug ; that forces single-thread applier even whenslave_parallel_threadsis non-zero.- Long master transactions (single
UPDATEtouching millions of rows, batchDELETEwithoutLIMIT) produce one huge binlog event that the applier MUST process serially. ALWAYS batch large DML on the master into chunks of 1000-10000 rows. binlog_row_image=MINIMALreduces binlog volume but RECORDS only the primary key in the before-image. Fix-on-replica row reconstruction and point-in-time recovery to a specific row state both BREAK withMINIMAL. NEVER setMINIMALon PITR-critical or multi-master fix-able instances.gtid_strict_mode=ONREFUSES any out-of-sequence GTID. A failed-over or partially-re-cloned replica with a stalegtid_slave_poswill stop withER_GTID_STRICT_OUT_OF_ORDERand silently grow lag until reset.- Semi-sync fallback to async (when
rpl_semi_sync_master_timeoutis reached) appears as a lag spike on monitoring graphs but is NOT a real applier-side lag. ALWAYS checkRpl_semi_sync_master_status(ONvsOFF) before treating a spike as applier-thread slowness. - Replica disk slower than master = permanent lag. NEVER pair a HDD replica with an SSD master for an OLTP workload ; the applier IO cannot catch up.