MySQL (Galera)

The cluster database is a three-node Percona XtraDB Cluster: persona-mysql in the percona-pxc-operator namespace, PXC 8.4, operator 1.19. Every write is replicated synchronously to all three nodes through Galera. That buys a database that survives losing a node without a failover, and it comes with rules a single MySQL server never had. Breaking them rarely fails the statement you typed. It breaks replication later, on another node, under someone else's query.

How traffic flows

Endpoint	Goes to	Use
persona-mysql-haproxy:3306	one node (pxc-0 while healthy)	every app, the exporter, the developer port-forward
persona-mysql-haproxy-replicas:3306	all three nodes, round-robin	read-only queries
scripts/db-import, scripts/db-shell, scripts/db-drop	pxc-0 directly, as root	imports, schema work, removing databases and users

One writer means there are no write conflicts between nodes, and session-level locks such as GET_LOCK() behave like on a single server.

Rules

InnoDB only. Galera replicates InnoDB and nothing else. A MyISAM or MEMORY table accepts writes on the node that receives them and on no other node, with nothing but a warning in the error log. The divergence stays invisible until something makes it replicable, and then a node is evicted. pxc_strict_mode=MASTER rejects DML on such tables.

Every table has a primary key. Galera applies row changes on the other nodes by primary key. Without one, an UPDATE or DELETE cannot be applied deterministically. MASTER mode rejects DML on tables without one.

DDL is cluster-wide and blocking. CREATE, ALTER, DROP, TRUNCATE, RENAME, CREATE USER, GRANT run under Total Order Isolation: every node executes the statement at the same point in the replication stream, and no other write anywhere in the cluster proceeds until it has finished on every node. A ten-second ALTER is a ten-second write freeze for every site. Prefer ALGORITHM=INSTANT or INPLACE where MySQL supports it, do table rewrites on large tables in quiet hours, and never put DDL in a loop. Frequent CREATE USER/GRANT in a loop is what wedged the whole cluster on 2026-07-24.

Imports go through scripts/db-import, copies through scripts/db-copy, exports through scripts/db-export. They stream through pxc-0 over one connection, dump with --single-transaction so the source is never locked, rewrite ENGINE=MyISAM to InnoDB on the way in, refuse to start unless all three nodes are synced, and list any non-InnoDB or primary-key-less table they leave behind. A GUI import over a port-forward does none of that.

Ad-hoc schema work goes through scripts/db-shell. Root on the write node, with strict mode active. The persona-* accounts over a port-forward are for looking at data.

Databases and users are dropped through scripts/db-drop. Crossplane orphans them when an app is removed; db-drop backs them up and drops them one statement at a time. See component-database.

Never convert a non-InnoDB table in place. Its rows differ per node, and ALTER TABLE ... ENGINE=InnoDB replicates that divergence. Swap in an empty InnoDB table atomically and rebuild the contents from the application:

CREATE TABLE craft_searchindex_new LIKE craft_searchindex;
ALTER TABLE craft_searchindex_new ENGINE=InnoDB;
RENAME TABLE craft_searchindex TO craft_searchindex_myisam,
             craft_searchindex_new TO craft_searchindex;
DROP TABLE craft_searchindex_myisam;

The RENAME is one atomic statement, so a write arriving during the swap waits on the metadata lock and lands in the InnoDB table. For Craft's craft_searchindex, rebuild afterwards with Utilities → Search Indexes in the control panel.

What strict mode rejects

pxc_strict_mode=MASTER is ENFORCING without the explicit-locking check. Rejected, with ERROR 1105 (HY000): Percona-XtraDB-Cluster prohibits ...:

• DML on a table whose engine is not InnoDB
• DML on a table without a primary key
• ALTER TABLE ... ENGINE=MyISAM (and other transactional-to-non-transactional changes)
• SERIALIZABLE transaction isolation, XA transactions, DISCARD/IMPORT TABLESPACE

Still allowed, because writes only ever hit one node: LOCK TABLES (mysqldump output), GET_LOCK()/RELEASE_LOCK() (Craft's database mutex, framework lock stores).

Configuration

Set in k8s/app/database/percona-mysql/config/cluster/percona-mysql.yaml. Changing configuration makes the operator roll the nodes one at a time.

Setting	Value	Why
pxc_strict_mode	MASTER	Rejects the things Galera cannot replicate while keeping LOCK TABLES and GET_LOCK() working.
innodb_buffer_pool_size	1536M	Dataset is several GB. An applier that has to read pages from disk falls behind and triggers flow control, which pauses writes on every node.
wsrep_applier_threads	4	Parallel apply on the two non-writing nodes.
wsrep_provider_options	gcache.size=2G	A node that is out for a few minutes (certificate rotation, node drain, operator roll) rejoins with an incremental transfer instead of copying the whole dataset.
max_connections	500	Not the bottleneck; Prisma clients need connection_limit in their URL.
wait_timeout	1800	Backstop for abandoned connections from crashed cron pods.
memory request / limit	3Gi / 4Gi	Buffer pool plus gcache, per-connection buffers and the operator's sidecars.

The persona-dev, persona-stg and persona-prod accounts and their wildcard grants live in MySQL itself, not in the CR: the operator cannot express a grant on dev-%.*, and its users field would re-run DDL on every reconcile. Each has SELECT, INSERT, UPDATE, DELETE, LOCK TABLES, SHOW VIEW, CREATE TEMPORARY TABLES ON <env>-%.* and nothing else, so a developer session can change data but never run DDL under Total Order Isolation. To change a grant, run REVOKE/GRANT through scripts/db-shell, one account at a time, with the cluster Synced.

Alerts

Alert	Fires when	What it means
MySQLNonInnoDBTable	a table with another engine exists for 10m	Something imported or created a table Galera cannot replicate. Swap it as described above before anyone alters it.
MySQLGaleraFlowControl	writes paused more than 10% of the time for 5m	A node cannot keep up. Usually a running import or a large ALTER.
MySQLGaleraNodeRestarted	a pxc container restarted	A node was evicted or crashed and is rejoining, usually via SST. Read its error log.
MySQLGaleraNotReady	wsrep_ready=0 for 5m	The node HAProxy is talking to has left the primary component.
MySQLGaleraClusterDegraded	fewer than 3 nodes for 10m	A node has not rejoined. At one node there is no quorum protection.

The mysqld error log is /var/lib/mysql/mysqld-error.log inside the pxc container, not container stdout. Root password: kubectl -n percona-pxc-operator exec persona-mysql-pxc-0 -c pxc -- cat /etc/mysql/mysql-users-secret/root.

What a node eviction looks like

Galera applies every writeset on every node. When one node fails to apply a writeset that the others applied, the nodes vote, and the minority leaves:

[Galera] Member 1(persona-mysql-pxc-1) initiates vote on ...:1779892: Can't find record in 'craft_searchindex', Error_code: 1032;
[Galera] Member 0(persona-mysql-pxc-2) initiates vote on ...:1779892: Can't find record in 'craft_searchindex', Error_code: 1032;
[ERROR] [Galera] Vote 0 (success) on ...:1779892 is inconsistent with group. Leaving cluster.

The evicted node fails its liveness probe, restarts, and takes a full state transfer (SST) from a donor. Around three minutes for the current dataset, during which the donor is also busy. HAProxy moves writes to the next node. Nothing needs to be done for the cluster; what needs doing is finding why that table differed between nodes, which is always one of the rules above.

What a TOI wedge looks like and how to recover

Different failure, different recovery. A DDL statement under Total Order Isolation can deadlock against a concurrent commit. After that every DDL on every node queues forever behind it, while most DML keeps flowing, so the symptom is clients that appear to hang on any ALTER, DROP DATABASE or CREATE USER. SHOW PROCESSLIST shows them in wsrep: preparing for TO isolation; KILL returns ERROR 1095.

A rolling restart does not fix it: a rejoining node needs an SST, and a wedged donor cannot serve one, so the joiner crash-loops. The fix is a controlled restart of all three nodes at once:

1. Stop whatever is producing DDL. For Crossplane-managed users and grants, annotate the user.mysql.sql.crossplane.io and grant.mysql.sql.crossplane.io resources with crossplane.io/paused=true. Scale the percona-pxc-operator deployment to 0 so it stops reconciling system users.
2. kubectl -n percona-pxc-operator delete pod persona-mysql-pxc-0 persona-mysql-pxc-1 persona-mysql-pxc-2 --grace-period=30. A wedged mysqld ignores SIGTERM and the default grace period is 600s.
3. The pods come back in crash-recovery mode, compare their sequence numbers and bootstrap from the most advanced one. Around six minutes of downtime.
4. Scale the operator back to 1 and remove the pause annotations once all three nodes report Synced.

Backups

The operator takes a daily XtraBackup at 02:00 to the mysql-backups-ax12d7kdcd bucket on OVH S3 and keeps 14.