Previewkernable is not released yet: these docs describe work in progress, and things may change before 1.0.Not released yet.Request early access →
Skip to content

kernable start ​

The kernel takes its settings as flags of kernable start. Every flag has an environment variable. A flag wins over its variable, and the variable over the default. Write a flag as --flag value or --flag=value; given twice, the last one wins, except --root, which adds a root each time. Relative paths resolve against the working directory. kernable start --help lists the flags. An unknown flag, a missing value or a malformed one ends kernable start with exit code 2 before the kernel starts.

sh
kernable start --root root.pub --store sqlite:kernable.db --listen 127.0.0.1:7233

The kernel holds no keys of services and no permissions: it trusts the root keys, and a root key says with a grant what each key may do (Trust, Manage keys). SIGHUP (Unix) reads the kernel key file again; every other change needs a restart.

--dev is for trying kernable out on one machine. It creates whatever is missing of root.key, kernel.key, vault.key, app.key and app.grant in the working directory, with a grant that lets app.key do everything. Unless you set them by flag or variable, it then uses --root root.pub, --vault-key vault.key and --ui-listen 127.0.0.1:7234. Every other flag keeps its default. In production, start with --root and grant each service only what it needs.

Logs go to stderr. KERNABLE_LOG sets the filter (default info) and KERNABLE_LOG_FORMAT=json writes JSON (Monitor kernable).

Trust ​

Which keys the kernel trusts, and its own key.

FlagVariableDefaultDescription
--rootKERNABLE_ROOTrequiredA root key the kernel trusts: its key ID (hex) or the path of its .pub file. Only a root key can sign grants and revocations. Repeat the flag for several roots, for example while you replace one; the variable takes them comma-separated. At least one is required; --dev uses root.pub if none is given.
--kernel-keyKERNABLE_KERNEL_KEYkernel.keyThe kernel's own signing key, written by kernable keys new kernel. It signs checkpoints and stored revocations. The file must exist. On Unix, SIGHUP reads it again, so a new key signs from then on without a restart.

Stores ​

Where the kernel keeps what. A store URL names a backend: sqlite:<path> (or a bare path), file:<path> or memory:; not every store takes every backend. memory: keeps nothing across a restart and is for tests. What to back up: Back up and restore.

FlagVariableDefaultDescription
--storeKERNABLE_STOREsqlite:kernable.dbThe event store. It also holds the command queue and results, checkpoints, revocations and registrations. sqlite:<path> or a bare path: SQLite. file:<path>: an append-only log file. memory:: nothing survives the process, for tests.
--durabilityKERNABLE_DURABILITYfullWhen a write to the event store counts as done. full: on the disk (fsync, a full flush on macOS) before a client, listener or worker hears of it, so a committed command survives a power loss or an OS crash. Commits waiting at the same time share one flush (group commit), so throughput holds up under concurrent load, while a single client waits about one flush per command. normal: handed to the OS, which survives a crash of the kernel but not of the machine, and is faster; --dev uses it unless given. The vault and the data service always write to the disk.
--snapshotsKERNABLE_SNAPSHOTS—The snapshot store, a cache of aggregate states that workers rebuild from the events. sqlite:<path> or a bare path: SQLite, kept across restarts. memory:: lost on a restart, so the first command per aggregate reads all its events again. Not file:. Default: SQLite next to the event store, <its name>-snapshots.db (kernable-snapshots.db for kernable.db), or memory: for a memory: store.
--vault-keyKERNABLE_VAULT_KEY—The key that encrypts the vault: a file with 32 random bytes, hex encoded (kernable keys new vault writes one). Without it the kernel has no vault: commands and events with vault values are refused. --dev uses vault.key if none is given.
--vault-storeKERNABLE_VAULT_STOREsqlite:vault.dbWhere the vault keeps personal and secret values, encrypted with --vault-key. sqlite:<path> or a bare path: SQLite. memory:: every value is lost on a restart, for tests. Not file:. Ignored without --vault-key.
--data-storeKERNABLE_DATA_STOREfile:dataWhere the data service keeps files. file:<directory>: one file per stored file, the directory is created with the first one. memory:: lost on a restart.
--cache-storeKERNABLE_CACHE_STOREmemory:Where the cache service keeps values. Only memory: exists: values last until they expire, are deleted, or the kernel restarts.

Network ​

Where the kernel listens. Plain TCP only on a loopback address; anything else needs mutual TLS (Set up TLS). The metrics and UI addresses serve plain HTTP with no login (Monitor kernable).

FlagVariableDefaultDescription
--listenKERNABLE_LISTEN127.0.0.1:7233The gRPC address for every service and client: host:port, or unix:<path> on Unix, whose owner and group may connect (mode 0660). Plain TCP is accepted only on a loopback address (127.0.0.1, ::1, localhost); any other address needs mutual TLS (--tls-cert, --tls-key, --tls-client-ca), or the kernel does not start.
--tls-certKERNABLE_KERNEL_TLS_CERT—Mutual TLS on a TCP --listen: the kernel's certificate (PEM). kernable certs new writes development certificates. Give all three TLS flags or none; TLS with a unix: address is an error.
--tls-keyKERNABLE_KERNEL_TLS_KEY—Mutual TLS: the private key of --tls-cert (PEM).
--tls-client-caKERNABLE_TLS_CLIENT_CA—Mutual TLS: the CA certificate (PEM) that must have signed every client's certificate. A client without one is refused.
--metrics-listenKERNABLE_METRICS_LISTEN—A TCP host:port for plain HTTP with no login: Prometheus metrics at GET /metrics, GET /health and GET /ready. Bind it to a private address. Off when empty. If the address cannot be bound, the kernel stops.
--ui-listenKERNABLE_UI_LISTEN—A TCP host:port for the kernel's UI in a browser: an overview of health and activity, the business processes and their cases, operations (services, lag, parked events, rejections, standby, store), and the domains with their commands, events and readers, the log and every aggregate's history. It has no login, so the address must be loopback, or the kernel does not start. If the address is taken, the kernel runs on without the UI and logs the UI is off. Off when empty; --dev sets 127.0.0.1:7234 unless it is given.

Timeouts and intervals ​

A timeout ends the wait, not the work: what happens then is in each row and in Failures and recovery.

FlagVariableDefaultDescription
--lease-ttl-msKERNABLE_LEASE_TTL_MS10000How long a worker holds a command, in milliseconds. If its result has not arrived by then, the command goes to another worker and a late result is ignored. A worker that disconnects gives its commands back at once. Set it above your slowest command handler.
--checkpoint-interval-msKERNABLE_CHECKPOINT_INTERVAL_MS5000How often, in milliseconds, the kernel signs a checkpoint of the hash chain with --kernel-key, if new events arrived. Events after the last checkpoint are not yet covered by a signature (kernable verify counts them).
--submit-timeout-msKERNABLE_SUBMIT_TIMEOUT_MS30000How long a submit waits for its command to be committed or rejected, in milliseconds. After that the client gets DEADLINE_EXCEEDED, but the command stays queued and still runs: resubmit it with the same command ID to get the result.
--settle-timeout-msKERNABLE_SETTLE_TIMEOUT_MS5000How long a settling submit (the default) waits after the commit for the listeners that count to process the events, in milliseconds. After that the client gets the committed outcome marked as not settled (NotSettled in the SDKs).
--query-timeout-msKERNABLE_QUERY_TIMEOUT_MS5000How long a query waits, in milliseconds, for a read model that answers it to catch up and answer. After that the client gets DEADLINE_EXCEEDED.
--middleware-timeout-msKERNABLE_MIDDLEWARE_TIMEOUT_MS5000How long a request waits for each middleware in its chain to catch up and give its verdict, in milliseconds. After that the request is refused with DEADLINE_EXCEEDED: middleware fails closed.
--drain-timeout-msKERNABLE_DRAIN_TIMEOUT_MS10000On SIGTERM or SIGINT, the kernel refuses new submits and queries and waits this long, in milliseconds, for running ones to finish before it exits. Commands still unfinished run again after a restart.

Limits ​

What the kernel takes on at most, so that no client, service or connection can exhaust its memory or capacity. Beyond a limit it refuses with RESOURCE_EXHAUSTED (a connection beyond --max-connections is closed) and counts the refusal in kernable_refused_total (Metrics). Every limit is at least 1.

FlagVariableDefaultDescription
--max-connectionsKERNABLE_MAX_CONNECTIONS1024Connections to the gRPC address at once. A connection beyond it is closed right away and counted in kernable_refused_total{reason="connections"}. Keep it below the process's limit of open files.
--max-streams-per-connectionKERNABLE_MAX_STREAMS_PER_CONNECTION256Requests and streams at once on one connection (HTTP/2 concurrent streams). Further ones wait in the client until one ends.
--max-message-bytesKERNABLE_MAX_MESSAGE_BYTES4194304The largest gRPC message the kernel receives or sends, in bytes; the SDKs' default is the same 4 MiB. The data service takes files up to 16 MiB regardless. A command whose snapshot and the events after it do not fit one message reaches the worker in parts; only a command with its snapshot, or a single event, that does not fit one message is rejected with too_large. Listener batches are cut to fit.
--handshake-timeout-msKERNABLE_HANDSHAKE_TIMEOUT_MS10000How long a worker or listener stream may take to send its first message and to answer the challenge, in milliseconds. A stream that is slower ends with DEADLINE_EXCEEDED.
--max-queued-commandsKERNABLE_MAX_QUEUED_COMMANDS100000Commands in the queue at once, waiting or running. A new command beyond it is refused with RESOURCE_EXHAUSTED and not stored; resubmitting one that is already stored is always answered.
--max-queued-per-aggregateKERNABLE_MAX_QUEUED_PER_AGGREGATE1000Commands in the queue at once for one aggregate. A new command beyond it is refused with RESOURCE_EXHAUSTED, so one busy aggregate cannot fill the queue.
--max-queued-bytesKERNABLE_MAX_QUEUED_BYTES536870912Bytes of commands in the queue at once (the queue lives in the kernel's memory). A new command beyond it is refused with RESOURCE_EXHAUSTED.
--max-requests-per-keyKERNABLE_MAX_REQUESTS_PER_KEY1000Submits, queries, file and cache requests of one key in progress at once. One beyond it is refused with RESOURCE_EXHAUSTED, so one service cannot take all the kernel's capacity.
--max-subscriptions-per-keyKERNABLE_MAX_SUBSCRIPTIONS_PER_KEY16Open event subscriptions (EventService.Subscribe) of one key at once. One beyond it is refused with RESOURCE_EXHAUSTED.
--max-listeners-per-keyKERNABLE_MAX_LISTENERS_PER_KEY64Listeners one key may register (connected or not) or run live at once. A new name beyond it is refused with RESOURCE_EXHAUSTED; registered ones always reconnect. A registered listener counts for settling and readiness, so this bounds how many a key can make everyone wait for.
--max-cache-bytesKERNABLE_MAX_CACHE_BYTES268435456Bytes the cache service keeps (values and keys). When a value would exceed it, the oldest values are dropped first, as a cache may; a single value larger than the limit is refused with RESOURCE_EXHAUSTED.
--max-data-bytesKERNABLE_MAX_DATA_BYTES16777216Bytes of one file of the data service, at most 134217728 (128 MiB). A larger put is refused with INVALID_ARGUMENT. A file travels whole in one message, so the kernel and the SDK hold it in memory while it is put or read.

Other ​

FlagVariableDefaultDescription
--replicationKERNABLE_REPLICATIONoffsync: a standby (--follow) holds every write before anyone is told of it. Commits, revocations, grants and memberships wait until the standby has stored them; without a standby in sync they wait, and /ready answers 503. Without a witness, until a standby has caught up once (a note beside the store keeps it), the kernel serves alone and the standby catches up meanwhile; only a standby told it caught up may be promoted. off: no standby.
--backupKERNABLE_BACKUPoffon: a backup follower (kernable backup, a key granted backup) may follow the kernel and write its records to object storage; the kernel keeps its record log for it, as for a standby. off: no backup follower.
--partitionKERNABLE_PARTITION—This kernel's partition, a name in the partition map: it serves the tenants the map assigns to it and refuses the others with FAILED_PRECONDITION wrong_partition, and none before it has a map. Without it, the kernel serves every tenant.
--controlKERNABLE_CONTROL—With --partition: the control partition's addresses, comma-separated (both kernels of a pair). This kernel takes the partition map, grants, revocations and memberships from there every second, with --node-key (granted replicate). Without it, this kernel is the control partition: maps are applied here (kernable partitions apply).
--witnessKERNABLE_WITNESS—Run as one kernel of a pair whose roles a witness (kernable witness) decides, at this address: the primary of the newest term serves with --replication sync, the other follows it, and takes its place on its own once the primary's lease runs out. Needs --node-key and --primary-kernel-key; not with --follow. The connection uses KERNABLE_TLS_*, as the SDKs do.
--node-keyKERNABLE_NODE_KEYnode.keyWith --witness or --control: this kernel's key towards the witness, the other kernel and the control partition, whose grant (<name>.grant next to it) allows replicate.
--advertiseKERNABLE_ADVERTISE—With --witness: the address the other kernel reaches this one at. Default: --listen.
--standby-timeout-msKERNABLE_STANDBY_TIMEOUT_MS5000With --witness: how long the primary's writes wait for a standby that is away or behind before the primary tells the witness and commits alone. A standby that missed writes is never made the primary.
--followKERNABLE_FOLLOW—Run as the standby of the primary at this address: apply its records, serve nothing else (every request is refused with UNAVAILABLE, not_primary) until kernable promote makes this kernel the primary. The connection uses KERNABLE_TLS_CA, KERNABLE_TLS_CERT and KERNABLE_TLS_KEY, as the SDKs do.
--follow-keyKERNABLE_FOLLOW_KEYstandby.keyWith --follow: the standby's key, whose grant (<name>.grant next to it) allows replicate.
--primary-kernel-keyKERNABLE_PRIMARY_KERNEL_KEY—With --follow: the primary's kernel key, as a key ID or a .pub file; the standby checks the primary's checkpoints with it. Repeat the flag for a key the primary had before; the variable takes them comma-separated.
--take-over-msKERNABLE_TAKE_OVER_MS0Start while another kernel still holds the stores, and take them over (a kernel update on the same machine): wait at most this long, in milliseconds, for it to let them go, then serve on the same address. The other kernel lets go when it gets SIGTERM, once its running requests finished. kernable node does this when its kernel's binary changes. 0: refuse stores another kernel holds.
--schedule-max-daysKERNABLE_SCHEDULE_MAX_DAYS366How far ahead, in days, a scheduled command (Command.deliver_at) may be due. One due later is refused with INVALID_ARGUMENT. Scheduled commands wait in the kernel's memory until due and count towards --max-queued-commands, --max-queued-per-aggregate and --max-queued-bytes.
--key-commandKERNABLE_KEY_COMMAND—A program and its arguments, separated by spaces, that unwraps a key: the kernel runs it with a key file's path as the last argument and reads the key from its output (64 hex characters, as in a key file), for the kernel key, the vault key and the node keys, at start and on SIGHUP. The key files then hold wrapped keys (age, a KMS, systemd-creds). Empty: the files hold the keys.
--max-store-bytesKERNABLE_MAX_STORE_BYTES0Bytes the event store, the vault and the files may take together, measured every 10 s (0: no limit). At the limit, new commands and file puts are refused with RESOURCE_EXHAUSTED quota_storage; queries, listeners, file deletes and the commands of --quota-exempt go on. At 80 % the kernel logs a warning.
--quota-exemptKERNABLE_QUOTA_EXEMPT—Commands --max-store-bytes never refuses, comma-separated (identity.ErasePerson, identity.*): list the commands that erase personal data, so an erasure request is always possible.
--max-requests-per-second-per-keyKERNABLE_MAX_REQUESTS_PER_SECOND_PER_KEY0Submits, queries, file and cache requests one key may start per second, with a burst of one second's worth (0: no limit). More are refused with RESOURCE_EXHAUSTED rate, to retry a moment later.