Replication (extension)
A replica group
A database may be kept by a group of servers, its replicas, each a node holding the whole database: one, the primary, takes the changes, and the others, the backups, hold what it commits and serve reads. A change is committed once a majority of the group holds it durably, so the group bears the loss of fewer than half its nodes: three nodes bear one lost, five bear two. When the primary is lost, the backups choose the next primary among themselves within seconds, and go on; the node lost, run again, rejoins and catches up. The protocol is Viewstamped Replication, each term of a primary a view.
Replication is set up on each node by its settings, given as the
server starts (--set, or a configuration file),
the node's own, kept with no database:
| Setting | Meaning |
|---|---|
| REPLICATION.REPLICA | this node's number among the members; 0, not replicated |
| REPLICATION.MEMBERS | every member: its number, =, and the host and port its replica listens at, by commas |
| REPLICATION.LISTEN | where this node listens for the others; its own member's host and port by default |
| REPLICATION.CA | the PEM file of the authorities the members' certificates are verified against |
| REPLICATION.TLS | ON: the members secure their frames by mutual TLS; OFF only on one machine |
| REPLICATION.TICK_MS | the pace of the protocol's clock: heartbeats, and how soon a lost primary is given up. A node has three ticks to connect to another, so the ticks must outlast the time a connection takes between members |
| REPLICATION.RETAIN_BYTES | how much of the group's log the node keeps before its last checkpoint, 256 MiB by default: a backup behind by less catches up from the log |
Each node presents SERVER.TLS_CERTIFICATE and SERVER.TLS_KEY to the others, and takes only a member whose certificate the authorities of REPLICATION.CA verify, its name the host REPLICATION.MEMBERS names for it. Three nodes:
$ turboreld --dir /var/db --keys keys --set REPLICATION.REPLICA=1 \
--set REPLICATION.MEMBERS=1=db1.example.com:7423,2=db2.example.com:7423,3=db3.example.com:7423 \
--set REPLICATION.CA=/etc/turborel/replicas-ca.pem \
--set SERVER.TLS_CERTIFICATE=/etc/turborel/db1.pem --set SERVER.TLS_KEY=/etc/turborel/db1.key
and the same on db2 and db3, each with its own number. A new
database is made by the group as its nodes first start, each over an
empty directory: give --owner NAME to each, which tells the node the
group is new, and the owner is made once a node serves, as a majority
of the nodes are up. The settings fixed when the database is made, and
every other setting, are the database's, replicated with it; the
REPLICATION settings alone are each node's. Give --owner only as the
group is made: a node started with it later over an empty directory
would make a database of its own, as a new group of one; a node started
over its files is made from them, whatever it is given.
A node starting over an empty log/, new to the group, or its log lost,
learns the group's state from the others before it takes part, told as
a recovering event: it joins once a majority of the members answer
it, the primary among them, and fetches the primary's database if its
own is behind. A node whose log/ directory is lost while its database's
files remain starts over from nothing, as the files cannot be recovered
without the log, and fetches the primary's database: nothing need be
copied by hand. A node started over an empty directory without
--owner waits so too, as it cannot tell a new group from one that
holds a log elsewhere: it is a new member, or one whose directory was
lost, and takes the group's state.
Serving
Clients connect to any node. The primary takes changes; a backup serves reads of what it holds, and refuses a change (SQLCODE -917, SQLSTATE 25000: the transaction's state). A read left open on a backup holds its snapshot, and the backup's checkpoints wait for it. A client whose statement is refused so connects to the primary, which INFORMATION_SCHEMA.REPLICAS names:
SELECT MEMBER, ADDRESS, IS_SELF, ROLE, EPOCH, VIEW_NUMBER, IS_CONNECTED, LAST_HEARD_MS
FROM INFORMATION_SCHEMA.REPLICAS;
MEMBER | ADDRESS | IS_SELF | ROLE | EPOCH | VIEW_NUMBER | IS_CONNECTED | LAST_HEARD_MS
--------+----------------------+---------+---------+-------+-------------+--------------+---------------
1 | db1.example.com:7423 | YES | PRIMARY | 0 | 0 | YES |
2 | db2.example.com:7423 | NO | BACKUP | 0 | 0 | YES | 18
3 | db3.example.com:7423 | NO | BACKUP | 0 | 0 | YES | 22
(3 rows)
It is the owner's to read. A commit on the primary returns once a majority holds the change: while fewer nodes are up, commits wait.
The replica's life is told to the diagnostics as replica events
(Diagnostics): the view entered and its primary,
the epoch entered and its members, this node serving, stopped or
removed, links to the other members up and down, and frames or
connections refused. The job that runs the replica is
turborel-replica (Background work).
Commands on a replicated database
A replicated database is managed through its primary, in SQL: users and
the owner by CREATE USER, ALTER USER, DROP USER and ALTER DATABASE OWNER
TO (Users), its files' formats by SET DATABASE
(Encryption), the group's members by ALTER DATABASE
REPLICATION MEMBERS, below. The commands on a database's directory,
user, reseal and rekey, refuse one that is replicated.
Changing the members
The group's members are changed while it runs, by the owner, on the primary (ALTER DATABASE):
ALTER DATABASE REPLICATION MEMBERS '1=db1.example.com:7423,2=db2.example.com:7423,4=db4.example.com:7423';
The literal names every member from then on, as REPLICATION.MEMBERS
does; here db3 leaves and db4 joins. The statement returns once the
group has committed the change, which ends the group's epoch: the
next epoch's primary is the first of the new members by number, so a
change that drops the primary, or adds a lower number, moves the
primary. The node's REPLICATION.MEMBERS setting is its starting point
only: the log carries each epoch's members, a node adopts the members
the log names over its setting, and tells the epoch it is in as an
epoch_entered event. INFORMATION_SCHEMA.REPLICAS shows the members of
the epoch the node is in, and its EPOCH.
To add a node, start its server, with settings naming the new members and its own number among them, once the statement has returned: it learns the group's state from the members as a new node does, and fetches the primary's database. A member's number names it for the life of the group and is never given again: the next node added is the next number.
A member that leaves answers the new members' requests for the state
until a majority of them are in the new epoch, then stops, told as a
removed event: its database serves nothing from then on, neither a
change nor a read, every statement refused (SQLCODE -917), until its
server is stopped. Its directory is of no use to the group any more.
Add or remove one member at a time, to a group whose members are all up and caught up: a group with a member recovering and another joining at once may not go on until one of them has caught up.
The log kept
Each node keeps the group's log only as far back as its own database
needs it: a checkpoint lets the records it holds go, but for
REPLICATION.RETAIN_BYTES of them, so a backup down for a while catches
up from the log of the others as it comes back. The log's segments in
log/ are dropped as records go. Each node syncs a record to log/
before it acknowledges it to the group, whatever LOG.DURABILITY says:
the setting bears on a database alone.
A node so far behind that the primary's log no longer reaches back to
what it holds fetches the primary's database instead: its files as the
primary's last checkpoint left them, with the log's records that
checkpoint is recovered with, over the same link, into transfer/ in
its directory, taking only the pages that differ from its own files and
copying the rest; then it installs them in place of its own, told as a
state_installed event, and follows the log from there. The primary
serves the transfer out of its files, which no checkpoint changes
meanwhile; a checkpoint due waits until the transfer is done. Nothing
stops on either node: the primary serves clients, and the node fetching
serves reads of what it held until the install. Started again with a
database far behind, a node fetches the state the same way.