Alpha

These pages are built from main, turborel 0.0.0. Anything may change from one build to the next: SQL, API, protocol and files alike.

Host Plug-ins

A host plug-in serves a point beneath the database. The pages it writes, the log it commits to, its clock, its files, its keys and the connections a server takes can all come from a program of another party. It speaks protocol 1, as an extension does, and is written with the same SDK (Plug-ins). The node declares it, never SQL, and starts it before the database is opened from its files. Each object it serves is used as the host's own.

A call crosses a pipe, so no host object is asked inside a commit's critical section, a latch or a lock. What the database needs there is a value it keeps, which a call filled before. Each section below states a kind's cost in calls, and choosing to pay it is the operator's decision.

Declaring a host plug-in

A host plug-in's manifest declares host objects and nothing else.

HOST <kind> <name> [ ( <option> = <number or 'text'>, ... ) ]
NAME VAULT; VERSION '1.0.0'; PROTOCOL 1; EXECUTABLE 'vault';
HOST ENCODING XOR (ID = 9001, OVERHEAD = 1);
HOST HASH FUNCTION XXH64 (ID = 9001, LENGTH = 8);
HOST LOG STORAGE FILELOG (BATCH = 256);
HOST CLOCK SYSTEM (EVERY = 10);

The setting PLUGINS.HOST names each host plug-in and its directory, separated by ;. A digest after @ pins its program's SHA-256, which a plug-in bringing package cryptography must have. The host sets it, in turboreld's configuration file or with --set, and SQL may not.

turboreld --dir /srv/db --set "PLUGINS.HOST=VAULT=/opt/vault"
listening at 127.0.0.1:5433

A program embedding the database sets it and starts its host plug-ins before it opens the files.

let db = Database::new();
db.set_setting("PLUGINS.HOST", &Value::Char("VAULT=/opt/vault".into()))?;
db.start_host_plugins()?;
db.open_files(&db.file_system()?, Path::new("/srv/db"), &OpenOptions::default())?;

INFORMATION_SCHEMA.HOST_PLUGINS shows the database's owner each object started, with its license as the manifest declares it, and turborel admin host-plugins lists them from the command line.

turborel admin host-plugins
PLUGIN_NAME  VERSION  OBJECT_TYPE  OBJECT_NAME  IS_PINNED  LICENSE  PROCESSES_RUNNING  CALLS  STOPPED
-----------  -------  -----------  -----------  ---------  -------  -----------------  -----  -------
VAULT        1.0.0    CLOCK        SYSTEM       NO         MIT                      1     52  NULL
(1 row)

Settings of a host plug-in

A host plug-in's manifest may declare settings as an extension's does (Plug-ins), named after the plug-in. They are NODE settings, which no other scope is allowed for, declared as the plug-in starts. Like PLUGINS.HOST they are the host's to give, in the configuration file, with --set or by Database::set_setting, and SQL may not set them. A configuration file may name them before they are declared, and a value that fits none is refused as the plug-in starts.

NAME VAULT; VERSION '1.0.0'; PROTOCOL 1; EXECUTABLE 'vault';
HOST BUFFER EVICTION KEEP (EVERY = 1000);
SETTING KEEP_PAGES TYPE INTEGER MIN 0 DEFAULT 0;
turboreld --dir /srv/db --set "PLUGINS.HOST=VAULT=/opt/vault" --set VAULT.KEEP_PAGES=64

The program reads them with each call, as an extension's program reads its own. INFORMATION_SCHEMA.HOST_PLUGINS lists each, of type SETTING.

When a host plug-in fails

A host object either answers or stops the node. A process that ends, breaks the protocol, sends too much or outlasts PLUGINS.CALL_TIMEOUT_MS stops it, and so does an error the kind does not take as an answer. Every statement after fails, and no page or record is written.

SELECT COUNT(*) FROM T;
ERROR 38T11: the host plug-in VAULT failed, so the node stopped until it starts again: the plug-in process ended: its output ended

turboreld stops serving and exits with the reason. At its next start the database is recovered from its log, as after a crash, and every commit made durable before is there.

Encodings and checksums

HOST ENCODING <name> (ID = <n>, OVERHEAD = <bytes>, CONFIDENTIAL = 0 | 1)
HOST HASH FUNCTION <name> (ID = <n>, LENGTH = <bytes>)

An encoding transforms each page and record the database writes. STORAGE.ENCODINGS names it, and the files it wrote name its ID, so a database is read through the program that wrote it. A page written is one ENCODE call and a page read one DECODE call, made with no latch held. A page that does not decode fails its read and the node goes on. A checksum is named by STORAGE.CHECKSUM and gives digests of its LENGTH, 1 to 64 bytes, one call a digest. The SDK's traits are PageEncoding and Checksum.

SET DATABASE STORAGE.ENCODINGS = 'XOR';
OK

The clock

HOST CLOCK <name> (EVERY = <milliseconds>)

The database asks the program its time every EVERY milliseconds, 10 if not written, one call each, and keeps the latest. CURRENT_TIMESTAMP and every commit read that copy, which never goes back. The SDK's trait is TimeSource.

The log and the catalog

HOST LOG STORAGE <name> (BATCH = <records>, DURABLE = 0 | 1)
HOST CATALOG STORAGE <name>

Log storage keeps the database's log. A commit appends in memory, and its sync sends every record not yet sent in one APPEND call, so the commits waiting together are made durable in one call. A commit whose sync the program fails is in doubt (08007) until the next start recovers. Recovery reads BATCH records a call, 256 if not written. Catalog storage records each image of the catalog in one call. The SDK's traits are LogStore and CatalogKeeper.

Spill

HOST SPILL <name>

A sort, a hash join or a grouping past its grant of MEMORY.WORK_BYTES spills to the program. Each spill area keeps one process while it lives. A run is pushed a batch at a time and read back 1024 records a call, and a table's read of a key is one call. The SDK's trait is SpillStore.

The file system

HOST FILE SYSTEM <name>

Every file the database opens, creates, removes or locks is the program's, named by its path. A page read or written is one call. Database::open, restore and turboreld open through it. The SDK's trait is Files.

Keys and package cryptography

HOST KEYRING <name>
HOST PACKAGE CRYPTO <name>

A keyring gives a 256-bit key by name, one call each, which encryption keeps while it reads with it. turboreld takes keys from the first keyring started, else from --keys. Package cryptography digests and verifies packages. Its program's digest must be pinned in PLUGINS.HOST, and the host's own cryptography checks it before each process starts. The SDK's traits are Keys and Crypto.

turboreld --dir /srv/db --set "PLUGINS.HOST=SIGNER=/opt/signer@3f8a...c41e"

Jobs

HOST BACKGROUND JOB <name>

A job runs as the database's own, labelled turborel-<name> and shown in INFORMATION_SCHEMA.JOBS. Each run is one call, whose answer says when to run again. The SDK's trait is Job.

SELECT LABEL FROM INFORMATION_SCHEMA.JOBS WHERE LABEL = 'turborel-tick' AND RUNS > 0;
LABEL
-------------
turborel-tick
(1 row)

Buffer eviction

HOST BUFFER EVICTION <name> [ ( EVERY = <milliseconds>, SAMPLE = <frames> ) ]

The pool chooses a page to evict on every miss, so it never calls the program to choose. The program instead sets the policy the pool's own evictor follows, which the pool reads with no lock. The evictor keeps, for each frame, its page, the intervals of its last two uses and its count of uses, where an interval passes each time a page is placed. A policy weighs these into one score, and the evictor takes the highest of a sample of frames.

Field Meaning
HISTORY 1 counts a page's age from its last use, least recently used; 2 from the use before, LRU-2
RECENCY the weight of age
FREQUENCY the weight taken off for each use, which keeps pages used often
ONCE added for a page used once, so a scan's pages go first
SAMPLE the frames each choice looks at, 32 if 0
ranges up to 64 ranges of a file's pages, each a weight, 100 being neutral

A range of weight 0 keeps its pages, which are taken only when no other frame looked at may be evicted. The SDK's trait is EvictionAdvisor, added by Plugin::host_eviction, and policies builds the common ones (lru, lru_2, scan_resistant, frequent, and weighing for a range).

The policy costs one call when the plug-in is started, before any pool is made, and one call each EVERY milliseconds after (1000 if it is not written), none on any read. That call carries the pool's report: the pages evicted since the last, at most 256, and SAMPLE frames (256 if it is not written), a window that moves on each time, with each frame's page, uses and last interval. The answer is a new policy, or none to keep the last. A process that dies stops the node, and until it stops the pool chooses by the last policy it was given.

Connections

HOST LISTENER <name>
HOST SERVED CONNECTIONS <name>

A listener takes connections beside SERVER.HOST and SERVER.SOCKET. Each connection keeps one process while it is open, so PLUGINS.PROCESSES bounds them. ACCEPT waits up to 50 milliseconds for a client and is asked again until one comes. The bytes pass in RELAY calls, one an exchange while the client talks and one each 50 milliseconds while it is quiet. No socket passes between processes, so the program may speak any transport to its clients. Served connections end and cancel connections by id, one call each. The SDK's traits are Accept and Connections.

What is no kind

The optimizer's rules are an extension's kind, over a plan form of their own (Optimizer Rules and Statistics). A wire protocol is a listener's to translate. Settings give the policies and the cost model, and the buffer pool's eviction is programmed by a BUFFER EVICTION object, not called. RULES.md D255 argues each.