Appearance
Back up and restore
Read this before you run a kernel with data you cannot lose. The event store is the system of record: everything else either depends on it (the vault, the keys) or is rebuilt from it (snapshots, read models). To copy a whole kernel into one file, for another machine or a self-hosted kernel, use export and import.
What to back up
| What | Flag (default) | Why | If lost |
|---|---|---|---|
| the event store | --store (sqlite:kernable.db) | commands, events, checkpoints, revocations, registrations of listeners and workers, applied memberships | everything is lost |
| the vault | --vault-store (sqlite:vault.db) | personal and secret values | personal data reads as null everywhere |
| the vault key | --vault-key | decrypts the vault | the vault cannot be read: same as losing it |
| files | --data-store (file:data) | files your processes put (Store files) | the files are gone |
| the kernel key | --kernel-key (kernel.key) | signs checkpoints and revocations | change it (Keys); keep every old public key for kernable verify |
| the root key | --root takes its .pub | signs grants and revocations | start with a new root and grant every key again (Keys) |
process keys and .grant files | — | sign requests; say what each key may do | create new keys and grant them with the root key |
| snapshots | --snapshots (sqlite:kernable-snapshots.db, next to the event store) | a cache | nothing: workers rebuild them, the first command per aggregate reads all its events |
| the cache | --cache-store (memory:) | values that expire | nothing: it is in memory and empty after every restart |
| read model stores | in your processes | caches of your views | nothing: read models rebuild from the events |
With kernable start --dev, all of these are in the working directory: kernable.db, kernable-snapshots.db, vault.db, data/, and the keys root, kernel, vault and app (.key and .pub), with app.grant. Where each kind of data lives and how long it is kept: Data over time.
Back up
Back up the event store, then the vault, right after each other. In that order, the worst case is a vault value that no event refers to yet, which does no harm.
With SQLite, use SQLite's online backup; it is safe while the kernel runs:
shsqlite3 kernable.db ".backup backup/kernable.db" sqlite3 vault.db ".backup backup/vault.db"Do not
cpa running SQLite store: the event store runs in WAL mode, and the latest commits are still inkernable.db-wal. To copy plain files, stop the kernel first.A
file:event log can be copied while the kernel runs. The copy may end in a torn record; the kernel cuts it off when it opens the copy and keeps the cut bytes in<path>.torn-<offset>.Copy the data directory (
data/by default). Each file is written next to its place and then renamed, so a copy never holds half a file.Check the backup of the event store:
shkernable verify sqlite:backup/kernable.db kernel.pubName every public key the kernel ever had. It works on a copy, without a running kernel:
text2311 events, 1903 commands, 41 checkpoints, 0 events after the last checkpoint OK0 eventsmeans the path is wrong:verifyopens a missing SQLite file as an empty store.OK, 1 suspect(exit code 2) means the log is intact but something was signed by a revoked key;TAMPERED(exit code 1) means it is not (Keys).
Keep keys apart from the data, as you would keep any secret, and the vault key apart from the vault backups. Keep the root key offline; it is needed only to issue grants and revocations.
Restore
- Stop the kernel.
- Put back the event store, the vault, the data directory, the keys and the
.grantfiles. - Delete the snapshot store if it is a SQLite file (
--snapshots sqlite:…). Snapshots newer than the restored log would not match it; workers rebuild them. - Check the store:
kernable verify <store> <kernel.pub>...must sayOK. - Start the kernel of the build that made the backup. A backup from an earlier development build does not carry over: before the first release, formats change in place and there are no migrations.
- Rebuild the read models that have a store (Rebuild a read model): their stores may be ahead of the restored log. Read models in memory read every event again on start anyway.
Everything written after the backup was taken is lost: commands, events, erasures, files.
Continuous backup
The steps above copy the stores now and then: what happened since the last copy is lost. A backup follower writes every record to object storage seconds after it is written, and kernable restore builds the stores again, up to the newest record or a point in time before something went wrong, such as a faulty deploy at 13:40.
Start the kernel with
--backup on. It keeps its record log for the follower, as for a standby.Make a key for the follower and a backup key:
shkernable keys new backup && kernable grant root.key backup.pub 'backup=*' kernable backup key backup-encryption.keyThe grant lets the key read every record and file; vault values reach it sealed with the kernel's vault key. The backup key encrypts everything the follower writes (AES-256-GCM): a bucket that leaks reveals nothing. Keep the backup key apart from the bucket, as you keep the root key; without it, no backup can be read.
Run the follower, next to the kernel or on another machine:
shAWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… KERNABLE_S3_ENDPOINT=https://fsn1.your-objectstorage.com \ kernable backup --follow https://10.0.0.1:7233,https://10.0.0.2:7233 --key backup.key \ --to s3://acme-backups/shop --backup-key backup-encryption.key--follownames both kernels of a pair: the follower turns to whichever is the primary.--toiss3://<bucket>/<prefix>for AWS S3 and S3-compatible storage (Hetzner, MinIO withKERNABLE_S3_PATH_STYLE=1), orfile:<directory>.KERNABLE_S3_ENDPOINTmay carry a path (https://storage.example.com/s3); a bucket name with dots needsKERNABLE_S3_PATH_STYLE=1over HTTPS, as its name does not match the endpoint's certificate otherwise. The connection usesKERNABLE_TLS_*, as the SDKs do.
What it writes, in generations: a base (a full image of the kernel, or nothing if the record log reaches back to the start), then a segment every --segment-seconds (default 5), 16 MiB or 4,096 records. The follower acknowledges records to the kernel only once their segment is in the target, so the kernel keeps them until then. A new generation starts every --base-hours (default 24), and the newest --keep (default 7) are kept. So a lost primary loses at most the last few seconds that were not uploaded yet, unless a standby holds them. The follower never holds up the kernel: it is not a standby, and a write never waits for it. Stopped and started again, it continues where its last segment ends; if the kernel no longer keeps those records, or its log differs from the backup's, a new generation begins.
kernable backup list s3://acme-backups/shop --backup-key backup-encryption.key shows the generations, each base's record and time, and how far its segments reach.
Restore from a continuous backup
sh
kernable restore s3://acme-backups/shop --backup-key backup-encryption.key \
--at 2026-11-04T13:39:00Z --store sqlite:kernable.db --vault-store sqlite:vault.db --vault-key vault.keyIt fills new, empty stores from the newest generation that begins before --at (or --at-record <n>; without either, everything), and stops before the first batch the follower took after that time. If the backup ends earlier, or holds nothing between the end of that generation and the base of the next, it says so in a note: line and restores up to there. Then it checks the store like kernable verify. --vault-key must be the source's vault key. Then start the kernel on the restored stores and rebuild the read models that have a store, as in steps 5 and 6 of Restore; the note on erasure below applies too.
Check a backup
A backup is only as good as its last restore. kernable backup check reads the newest generation all through without writing a store: every object opens with the backup key and decodes, the base holds the records its marker names, and the segments' records chain from the base's hash to the last one without a gap. It prints the record a restore would reach and how old it is, and exits 1 if the backup does not check out:
sh
kernable backup check s3://acme-backups/shop --backup-key backup-encryption.keyThe follower runs it by itself with --check-hours <h> (check-hours in a node's [backup]) and logs the backup checks out or, as an error, why not: alert on that line. Restore into scratch stores now and then as well (bench/backup-drill.sh does it end to end): a check proves the bytes, a restore also the time it takes.
What it loses, and how long the way back takes
Measured with bench/backup-drill.sh on a 2-core cloud server (Hetzner, 2026-11-07), the backup in object storage in the same region: a kernel on a store of 200,000 commands takes 200 commands a second, and at a random moment the kernel, the follower and the clients are killed together.
| Segments every 5 s | Segments every 1 s | |
|---|---|---|
| Commands stored but not in the backup | 126 (0.6 s); 759 (3.8 s) in another run | 127 (0.6 s) |
| Restore of about 206,000 events, 250 MB, verify included | 63 s | 68 s |
| From the restore's start until a kernel on it answers | 64 s | 69 s |
A store five times as large (1,006,043 events, 1.17 GB) lost 559 commands (2.8 s) and was back after 319 s: the way back grows with the store, about linearly. So the point in time a restore reaches is at most --segment-seconds and an upload behind the loss. The restore's time grows with the store: most of it goes to applying the records and checking every signature (on every core). A standby loses nothing a client was told and takes over in seconds; the backup is for the machine and the standby lost together, for mistakes, and for keeping history elsewhere.
Backups and erasure
A backup keeps personal data that was erased after it was taken. Keep vault backups only as long as your retention rules allow.
The kernel does not repeat erasures by itself. If you restore a vault that is older than the event store, or older than an erasure that happened since, delete the erased values again. The log names every erased subject in the erases column of its erasing events. With the kernel stopped:
sh
sqlite3 vault.db "ATTACH 'kernable.db' AS log;
DELETE FROM vault WHERE secret = 0 AND EXISTS (
SELECT 1 FROM log.events e WHERE e.erases = vault.subject AND e.tenant_id = vault.tenant
AND e.created_at >= vault.created_at);"This deletes the values of each erased subject that were written up to its erasure, and nothing written later. Then delete the snapshot store, as in step 3 above.
Files are never erased with a person; delete them yourself (Handle an erasure request).
