Appearance
Keys per service, and changing a key
Use this page when you move from the tutorial's single key to production, give a new service its rights, or replace a key: routinely, or because it leaked. How keys and grants work is in Trust.
| Situation | What to do | Kernel restart? |
|---|---|---|
| a new service, or more rights | issue a grant | no |
| fewer rights | change the key, revoke the old one | no |
| a key leaked | revoke it | no |
| the kernel's key | replace the file, SIGHUP | no |
| the vault's key | re-encrypt the vault | yes |
| the root key | start with both roots | yes, twice |
One key per service
The tutorial uses one key for everything (--dev writes app.key with a grant for everything). In production, give every service its own key and grant it only what it needs. A root key signs the grants; the kernel trusts root keys only.
Create the keys. Each call writes
<name>.key(secret) and<name>.pub(the key ID):shfor k in root api confirmations shop-worker read-models deploy; do kernable keys new keys/$k; doneKeep
keys/root.keyoffline; the kernel needs onlykeys/root.pub.Grant each key what it needs:
shkernable grant keys/root.key keys/api.pub \ submit=shop.Register,shop.Place,shop.AddItem,shop.Submit 'query=shop.*' kernable grant keys/root.key keys/confirmations.pub \ submit=shop.Confirm listen=shop.OrderSubmitted query=shop.GetOrder,shop.GetCustomer kernable grant keys/root.key keys/shop-worker.pub 'run=shop.*' kernable grant keys/root.key keys/read-models.pub 'listen=shop.*' 'answer=shop.*' 'vault_read=shop.*' kernable grant keys/root.key keys/deploy.pub manage=shop # applies the shop's membershipEach call prints the permissions and
wrote keys/<name>.grant. A permission is<action>=<type>,<type>, or a bare type for every action; a type isshop.Place,shop.*or*. The actions are in Trust;managetakes service names.Start the kernel with the root's public key:
kernable start --root keys/root.pub ....Give each service its
.keyand.grantfile. The SDKs read the.grantfile next to the key file, or the file in thegrantoption orKERNABLE_GRANT, and send it with every call. The key file comes from thekeyoption orKERNABLE_KEY(defaultapp.key).
A process that runs an app (kernable.app(...).run()) applies its service's membership on every start, so its key also needs manage=<service> (Deploy and upgrade). Without that, a single deploy key applies it.
A key without a grant, or whose grant does not cover a type, gets PERMISSION_DENIED. A grant takes effect when a process first presents it: the kernel's settings hold no keys of services, so a new service needs no change to the kernel.
Changing what a key may do
Issue a new grant for the key and restart the service with it. The kernel keeps the newest grant per key: once it has seen the new one, it refuses the old one, and workers, listeners and subscriptions whose key may no longer do what it does are disconnected. A listener or subscription that asked for every event type stays connected and gets only the types the new grant allows. A new service needs only its grant; the kernel does not change.
A grant cannot be withdrawn, only replaced by a newer one, which the kernel keeps across restarts. To take rights away from a key that may be stolen, change the key and revoke the old one.
Changing a key
Signatures in the log stay verifiable after a change: a key ID is the public key itself.
A client or worker key: create the new key, grant it what the old one had, and restart the process with the new key and grant. Revoke the old key if it may have leaked.
A listener's key is part of its registration. Grant the new key with --succeeds <old key>, and the listener keeps its position, order and middleware registration:
sh
kernable keys new keys/read-models-2
kernable grant keys/root.key keys/read-models-2.pub --succeeds keys/read-models.pub \
'listen=shop.*' 'answer=shop.*' 'vault_read=shop.*'Then restart the listener with the new key and grant. Without succeeds, another key is refused, so no service can take over another's listener.
Revoking a stolen key
When a key leaks, revoke it from the moment it may have been used. The root key signs the revocation, and the running kernel takes it at once:
sh
kernable revoke http://127.0.0.1:7233 keys/root.key keys/api.pub 2026-10-27T09:00:00Z # UTCIt answers revoked <key ID>. kernable revoke connects over mutual TLS when KERNABLE_TLS_CA, KERNABLE_TLS_CERT and KERNABLE_TLS_KEY are set, like the SDKs. The revoke request is valid for 60 s around its creation. The kernel then:
- stores the revocation, signed with its own key; it is never undone and holds after a restart;
- refuses the key everywhere, also with a grant, and disconnects its sessions;
- rejects commands of the key that still wait in the queue and were accepted after that time (
key_revoked).
kernable verify then lists everything the key signed from that time on:
text
$ kernable verify sqlite:data/shop.db keys/kernel.pub
2311 events, 1903 commands, 41 checkpoints, 0 events after the last checkpoint
SUSPECT command c-8f2a: signed by revoked key 1f0a…
OK, 1 suspectIt exits with 2 when there are suspects, 1 when the log was tampered with. Review what the suspects did, and undo it with new commands where needed: the log itself is never rewritten.
The kernel's key
The kernel's key signs checkpoints and revocations. Replace it without a restart: write the new key to the file --kernel-key names, and send the kernel SIGHUP (Unix only). kernable keys new never overwrites a key, so create it under another name and move it:
sh
cp keys/kernel.pub keys/kernel-old.pub # verify needs it later
kernable keys new keys/kernel-2
mv keys/kernel-2.key keys/kernel.key && mv keys/kernel-2.pub keys/kernel.pub
kill -HUP <pid>The kernel logs kernel key read again, and signs with the new key from then on. If it cannot read the file, it logs an error and keeps the previous key. SIGHUP changes nothing else; other settings need a restart.
Keep every old public key: kernable verify needs every key the kernel ever signed with, kernable verify <store> keys/kernel-old.pub keys/kernel.pub.
The vault's key
Stop the kernel, encrypt the vault again, and start it with --vault-key pointing to the new file:
sh
kernable keys new keys/vault-2 >/dev/null
kernable vault rekey sqlite:data/vault.db keys/vault.key keys/vault-2.keyThe old key cannot read the vault afterwards. Backups made before still need the old key.
Rekey also when a key has encrypted many values. Each value gets a random nonce, and one key is safe for 2^32 of them. The kernel counts them, logs a warning at startup past 2^31 (the UI's verdict warns too, and Explore → Data → Vault shows the count), and refuses new values at 2^32 until the vault is rekeyed.
Keys that are not on the disk in clear
With --key-command, the key files of the kernel, the vault and the node hold wrapped keys, and a program you name unwraps them when the kernel starts (and on SIGHUP). The kernel runs it with the key file's path as its last argument and reads the key from its output: 64 hex characters, as a key file holds them. Nothing in the kernel depends on a provider; any tool that decrypts a file works.
With age and an identity on a hardware key or in a protected place:
sh
age -e -r age1… -o kernel.key.age kernel.key && shred -u kernel.key
kernable start --kernel-key kernel.key.age --key-command "age -d -i /etc/kernable/age.identity" …With systemd-creds, bound to the machine's TPM:
sh
systemd-creds encrypt --with-key=tpm2 kernel.key kernel.key.cred && shred -u kernel.key
kernable start --kernel-key kernel.key.cred --key-command "systemd-creds decrypt" …With a cloud KMS, a small script that takes the path, for example /usr/local/bin/kms-unwrap:
sh
#!/bin/sh
aws kms decrypt --ciphertext-blob "fileb://$1" --query Plaintext --output text | base64 -dThe command runs once per key file and has 30 s for it; the kernel keeps the unwrapped key in memory until SIGHUP, when it runs the command again. A key the command cannot unwrap stops the kernel at start, with the command's message; on SIGHUP the previous key stays. A key bound to one machine (the TPM) needs wrapping again for the standby on another one. TLS keys (--tls-key) are read as files.
The root key
The root key signs only grants and revocations, so it can stay offline: issue grants on a separate machine and copy the .grant files. To replace it, start the kernel with both roots (--root keys/root.pub --root keys/root-2.pub), issue new grants with the new root and restart the services with them. Then restart the kernel without the old --root: grants signed by it are refused from then on. If the old root leaked, revoke it first, signed with the new root (kernable revoke <addr> keys/root-2.key keys/root.pub <since>): the kernel drops the grants the old root signed at once and refuses them from then on.
