# kernable documentation Every page of https://kernable.com/docs, in the order of its sidebar. For a short guide to building on kernable, see https://kernable.com/docs/kernable.AGENTS.md. **Status:** kernable is in private preview: not released yet, the install scripts and SDK packages are not public, and the API may change before 1.0. --- # Getting started In this guide you run a kernel, write business logic and a read model in TypeScript, Python or Rust, in the same language or two different ones, and see that a request reads its own writes. Then you run the reference application. You need Node.js ≥ 22.18 (step 5 submits from TypeScript), and Python ≥ 3.10 or Rust (stable) for the steps you write in them. ## 1. Install the kernel kernable is in private preview: the install scripts and the SDK packages (`@kernable/sdk` on npm, `kernable-sdk` on PyPI and crates.io) are not public yet, so the commands below do not work for everyone yet. [Request early access](https://kernable.com/#access) to get the binaries and the SDKs. ```sh [macOS, Linux] curl -fsSL https://kernable.com/install.sh | sh ``` ```powershell [Windows] irm https://kernable.com/install.ps1 | iex ``` This puts `kernable` on your `PATH`: `kernable start` runs the kernel, the other commands create keys and grants and verify a log (`kernable` alone lists them). ## 2. Start a kernel ```sh mkdir kernable-demo cd kernable-demo kernable start --dev ``` The kernel keeps its events in `kernable.db` in this directory and listens on `http://127.0.0.1:7233`, for this machine only. `--dev` also creates the keys this guide uses, among them `app.key`, which may do everything here. [Keys and grants](#keys-and-grants) below explains what that means. `--dev` also turns on the kernel's UI: open **http://127.0.0.1:7234** in a browser. It stays empty until the next steps connect something; then it shows them live. Leave the kernel running and continue in a second terminal. ## 3. Business logic A shopping cart: commands add items and check it out; events record what happened, and the cart's state is rebuilt from them. Pick TypeScript, Python or Rust; the tabs switch together. **1. Register the aggregate** with its initial state: TypeScript: ```ts const shopDomain = domain("shop"); const cartAggregate = shopDomain.aggregate("cart", { items: 0, checkedOut: false }); ``` Python: ```python shop_domain = domain("shop") cart_aggregate = shop_domain.aggregate("cart", {"items": 0, "checked_out": False}) ``` Rust: ```rust #[derive(Clone, Default, Serialize, Deserialize)] struct Cart { items: u32, checked_out: bool, } let shop_domain = domain("shop"); let cart_aggregate = shop_domain.aggregate("cart", Cart::default()); ``` **2. Apply events** to the state: TypeScript: ```ts cartAggregate.apply("ItemAdded", (state, data: { qty: number }) => { state.items += data.qty; }); cartAggregate.apply("CheckedOut", (state) => { state.checkedOut = true; }); ``` Python: ```python @cart_aggregate.apply("ItemAdded") def _(state, data): state["items"] += data["qty"] @cart_aggregate.apply("CheckedOut") def _(state, data): state["checked_out"] = True ``` Rust: ```rust #[derive(Serialize, Deserialize)] struct Item { sku: String, qty: u32, } cart_aggregate.apply("ItemAdded", |state, data: Item, _evt, _ctx| { state.items += data.qty }); cart_aggregate.apply("CheckedOut", |state, _: Value, _evt, _ctx| { state.checked_out = true }); ``` **3. Decide commands:** return events, or reject with `Rejected`: TypeScript: ```ts cartAggregate.command("AddItem", (state, data: { sku: string; qty: number }) => { if (state.checkedOut) throw new Rejected("Cart is checked out"); return [{ type: "ItemAdded", data }]; }); cartAggregate.command("CheckOut", (state, _data, _cmd, ctx) => { if (state.items === 0) throw new Rejected("Cart is empty"); // ctx.now() and ctx.id() stay the same if the command runs again. return [{ type: "CheckedOut", data: { at: ctx.now().toISOString(), orderId: ctx.id() } }]; }); ``` Python: ```python @cart_aggregate.command("AddItem") def _(state, data): if state["checked_out"]: raise Rejected("Cart is checked out") return [{"type": "ItemAdded", "data": data}] @cart_aggregate.command("CheckOut") def _(state, data, cmd, ctx): if state["items"] == 0: raise Rejected("Cart is empty") # ctx.now() and ctx.id() stay the same if the command runs again. return [{"type": "CheckedOut", "data": {"at": ctx.now().isoformat(), "orderId": ctx.id()}}] ``` Rust: ```rust cart_aggregate.command("AddItem", |state, data: Item, _cmd, _ctx| { if state.checked_out { return Err(Rejected::new("Cart is checked out")); } Ok(vec![NewEvent::new("ItemAdded", data)]) }); cart_aggregate.command("CheckOut", |state, _: Value, _cmd, ctx| { if state.items == 0 { return Err(Rejected::new("Cart is empty")); } // ctx.now() and ctx.id() stay the same if the command runs again. Ok(vec![NewEvent::new( "CheckedOut", json!({ "at": ctx.now().to_string(), "orderId": ctx.id() }), )]) }); ``` **4. Run a worker** that executes the cart's commands: TypeScript: ```ts const kernable = new Kernable({ key: "app.key" }); await kernable.worker(cartAggregate).run(); ``` Python: ```python k = Kernable(key="app.key") k.worker(cart_aggregate).run() ``` Rust: ```rust let kernable = Kernable::new("app.key")?; kernable.worker(&cart_aggregate).run().await; ``` In a second terminal, in `kernable-demo`, set up a project, save the whole file and run it. Leave it running; it prints where it connected and what it runs: ```text 12:04:10 worker connected to http://127.0.0.1:7233 · key app (69212a19f36f…) · runs shop.AddItem, shop.CheckOut ``` ```sh [TypeScript] npm init -y npm pkg set type=module npm install @kernable/sdk # save cart.ts, then (Node ≥ 22.18 runs TypeScript directly): node cart.ts ``` ```sh [Python] python3 -m venv py # Windows: python -m venv py py/bin/pip install kernable-sdk # Windows: py\Scripts\pip install kernable-sdk # save cart.py, then: py/bin/python cart.py # Windows: py\Scripts\python cart.py ``` ```sh [Rust] cargo init --name demo cargo add kernable-sdk serde_json cargo add serde --features derive cargo add tokio --features macros,rt-multi-thread # save src/bin/cart.rs, then: cargo run --bin cart ``` cart.ts: ```ts import { Kernable, Rejected, domain } from "@kernable/sdk"; const shopDomain = domain("shop"); const cartAggregate = shopDomain.aggregate("cart", { items: 0, checkedOut: false }); cartAggregate.apply("ItemAdded", (state, data: { qty: number }) => { state.items += data.qty; }); cartAggregate.apply("CheckedOut", (state) => { state.checkedOut = true; }); cartAggregate.command("AddItem", (state, data: { sku: string; qty: number }) => { if (state.checkedOut) throw new Rejected("Cart is checked out"); return [{ type: "ItemAdded", data }]; }); cartAggregate.command("CheckOut", (state, _data, _cmd, ctx) => { if (state.items === 0) throw new Rejected("Cart is empty"); // ctx.now() and ctx.id() stay the same if the command runs again. return [{ type: "CheckedOut", data: { at: ctx.now().toISOString(), orderId: ctx.id() } }]; }); const kernable = new Kernable({ key: "app.key" }); await kernable.worker(cartAggregate).run(); ``` cart.py: ```python from kernable import Kernable, Rejected, domain shop_domain = domain("shop") cart_aggregate = shop_domain.aggregate("cart", {"items": 0, "checked_out": False}) @cart_aggregate.apply("ItemAdded") def _(state, data): state["items"] += data["qty"] @cart_aggregate.apply("CheckedOut") def _(state, data): state["checked_out"] = True @cart_aggregate.command("AddItem") def _(state, data): if state["checked_out"]: raise Rejected("Cart is checked out") return [{"type": "ItemAdded", "data": data}] @cart_aggregate.command("CheckOut") def _(state, data, cmd, ctx): if state["items"] == 0: raise Rejected("Cart is empty") # ctx.now() and ctx.id() stay the same if the command runs again. return [{"type": "CheckedOut", "data": {"at": ctx.now().isoformat(), "orderId": ctx.id()}}] k = Kernable(key="app.key") k.worker(cart_aggregate).run() ``` src/bin/cart.rs: ```rust use kernable::{BoxError, Kernable, NewEvent, Rejected, domain}; use serde::{Deserialize, Serialize}; use serde_json::{Value, json}; #[tokio::main] async fn main() -> Result<(), BoxError> { #[derive(Clone, Default, Serialize, Deserialize)] struct Cart { items: u32, checked_out: bool, } let shop_domain = domain("shop"); let cart_aggregate = shop_domain.aggregate("cart", Cart::default()); #[derive(Serialize, Deserialize)] struct Item { sku: String, qty: u32, } cart_aggregate.apply("ItemAdded", |state, data: Item, _evt, _ctx| { state.items += data.qty }); cart_aggregate.apply("CheckedOut", |state, _: Value, _evt, _ctx| { state.checked_out = true }); cart_aggregate.command("AddItem", |state, data: Item, _cmd, _ctx| { if state.checked_out { return Err(Rejected::new("Cart is checked out")); } Ok(vec![NewEvent::new("ItemAdded", data)]) }); cart_aggregate.command("CheckOut", |state, _: Value, _cmd, ctx| { if state.items == 0 { return Err(Rejected::new("Cart is empty")); } // ctx.now() and ctx.id() stay the same if the command runs again. Ok(vec![NewEvent::new( "CheckedOut", json!({ "at": ctx.now().to_string(), "orderId": ctx.id() }), )]) }); let kernable = Kernable::new("app.key")?; kernable.worker(&cart_aggregate).run().await; Ok(()) } ``` More on aggregates and their handlers: [Event sourcing](../concepts/event-sourcing). ## 4. A read model A read model is a **listener**: it builds its own view from the events and answers **queries** from it. Queries always go through the kernel, so signatures and grants apply to every read. The read model doesn't have to be in the worker's language: pick any tab, independent of step 3. **1. Register the read model** on the domain, like the aggregate: its event and query types are then the domain's (`ItemAdded`, `GetCart`); another domain's would be qualified (`billing.InvoicePaid`). It keeps its rows in memory and reads every event again when it starts; with a store (`sqlite:carts.db`) it would keep them in a file and continue where it left off: TypeScript: ```ts const shopDomain = domain("shop"); const cartsReadmodel = shopDomain.readModel("carts"); const rows = new Map(); ``` Python: ```python rows = {} shop_domain = domain("shop") carts_readmodel = shop_domain.read_model("carts") ``` Rust: ```rust #[derive(Clone, Default, Serialize)] struct Row { items: u32, #[serde(skip_serializing_if = "Option::is_none")] order: Option, } let rows = Arc::new(Mutex::new(HashMap::::new())); let shop_domain = domain("shop"); let carts_readmodel = shop_domain.read_model("carts"); ``` **2. Handle events** and update the rows: TypeScript: ```ts cartsReadmodel.on("ItemAdded", (data: { qty: number }, evt) => { const row = rows.get(evt.aggregateId) ?? { items: 0 }; row.items += data.qty; rows.set(evt.aggregateId, row); }); cartsReadmodel.on("CheckedOut", (data: { orderId: string }, evt) => { rows.get(evt.aggregateId)!.order = data.orderId; }); ``` Python: ```python @carts_readmodel.on("ItemAdded") def _(data, evt): rows.setdefault(evt.aggregate_id, {"items": 0})["items"] += data["qty"] @carts_readmodel.on("CheckedOut") def _(data, evt): rows[evt.aggregate_id]["order"] = data["orderId"] ``` Rust: ```rust #[derive(Deserialize)] struct ItemAdded { qty: u32, } #[derive(Deserialize)] #[serde(rename_all = "camelCase")] struct CheckedOut { order_id: String, } let r = Arc::clone(&rows); carts_readmodel.on("ItemAdded", move |data: ItemAdded, evt, _ctx| { r.lock().unwrap().entry(evt.aggregate_id).or_default().items += data.qty; async { Ok(()) } }); let r = Arc::clone(&rows); carts_readmodel.on("CheckedOut", move |data: CheckedOut, evt, _ctx| { r.lock().unwrap().entry(evt.aggregate_id).or_default().order = Some(data.order_id); async { Ok(()) } }); ``` **3. Answer queries** from the rows: TypeScript: ```ts cartsReadmodel.query("GetCart", (data: { id: string }) => { const row = rows.get(data.id); if (!row) throw new Rejected("no such cart", "not_found"); return row; }); ``` Python: ```python @carts_readmodel.query("GetCart") def _(data): if data["id"] not in rows: raise Rejected("no such cart", "not_found") return rows[data["id"]] ``` Rust: ```rust #[derive(Deserialize)] struct GetCart { id: String, } let r = Arc::clone(&rows); carts_readmodel.query("GetCart", move |data: GetCart, _qry, _ctx| { let row = r.lock().unwrap().get(&data.id).cloned(); async move { Ok(row.ok_or_else(|| Rejected::new("no such cart").code("not_found"))?) } }); ``` **4. Run it** through the connection, as the worker runs the aggregate. Read models run before reactors (order 0 and 10), so reactors can query them: TypeScript: ```ts const kernable = new Kernable({ key: "app.key" }); await kernable.listener(cartsReadmodel).run(); ``` Python: ```python k = Kernable(key="app.key") k.listener(carts_readmodel).run() ``` Rust: ```rust let kernable = Kernable::new("app.key")?; kernable.listener(&carts_readmodel).run().await; ``` In a third terminal, in `kernable-demo`, set up the language as in step 3 if it is new to you, save the whole file and run it. It prints where it starts and what it reads and answers: ```text 12:04:31 listener carts connected to http://127.0.0.1:7233 · key app (69212a19f36f…) · from 0 · reads shop.ItemAdded, shop.CheckedOut · answers shop.GetCart ``` ```sh [TypeScript] node carts.ts ``` ```sh [Python] py/bin/python carts.py # Windows: py\Scripts\python carts.py ``` ```sh [Rust] cargo run --bin carts # saved as src/bin/carts.rs ``` carts.ts: ```ts import { Kernable, Rejected, domain } from "@kernable/sdk"; const shopDomain = domain("shop"); const cartsReadmodel = shopDomain.readModel("carts"); const rows = new Map(); cartsReadmodel.on("ItemAdded", (data: { qty: number }, evt) => { const row = rows.get(evt.aggregateId) ?? { items: 0 }; row.items += data.qty; rows.set(evt.aggregateId, row); }); cartsReadmodel.on("CheckedOut", (data: { orderId: string }, evt) => { rows.get(evt.aggregateId)!.order = data.orderId; }); cartsReadmodel.query("GetCart", (data: { id: string }) => { const row = rows.get(data.id); if (!row) throw new Rejected("no such cart", "not_found"); return row; }); const kernable = new Kernable({ key: "app.key" }); await kernable.listener(cartsReadmodel).run(); ``` carts.py: ```python from kernable import Kernable, Rejected, domain rows = {} shop_domain = domain("shop") carts_readmodel = shop_domain.read_model("carts") @carts_readmodel.on("ItemAdded") def _(data, evt): rows.setdefault(evt.aggregate_id, {"items": 0})["items"] += data["qty"] @carts_readmodel.on("CheckedOut") def _(data, evt): rows[evt.aggregate_id]["order"] = data["orderId"] @carts_readmodel.query("GetCart") def _(data): if data["id"] not in rows: raise Rejected("no such cart", "not_found") return rows[data["id"]] k = Kernable(key="app.key") k.listener(carts_readmodel).run() ``` src/bin/carts.rs: ```rust use std::{ collections::HashMap, sync::{Arc, Mutex}, }; use kernable::{BoxError, Kernable, Rejected, domain}; use serde::{Deserialize, Serialize}; #[tokio::main] async fn main() -> Result<(), BoxError> { #[derive(Clone, Default, Serialize)] struct Row { items: u32, #[serde(skip_serializing_if = "Option::is_none")] order: Option, } let rows = Arc::new(Mutex::new(HashMap::::new())); let shop_domain = domain("shop"); let carts_readmodel = shop_domain.read_model("carts"); #[derive(Deserialize)] struct ItemAdded { qty: u32, } #[derive(Deserialize)] #[serde(rename_all = "camelCase")] struct CheckedOut { order_id: String, } let r = Arc::clone(&rows); carts_readmodel.on("ItemAdded", move |data: ItemAdded, evt, _ctx| { r.lock().unwrap().entry(evt.aggregate_id).or_default().items += data.qty; async { Ok(()) } }); let r = Arc::clone(&rows); carts_readmodel.on("CheckedOut", move |data: CheckedOut, evt, _ctx| { r.lock().unwrap().entry(evt.aggregate_id).or_default().order = Some(data.order_id); async { Ok(()) } }); #[derive(Deserialize)] struct GetCart { id: String, } let r = Arc::clone(&rows); carts_readmodel.query("GetCart", move |data: GetCart, _qry, _ctx| { let row = r.lock().unwrap().get(&data.id).cloned(); async move { Ok(row.ok_or_else(|| Rejected::new("no such cart").code("not_found"))?) } }); let kernable = Kernable::new("app.key")?; kernable.listener(&carts_readmodel).run().await; Ok(()) } ``` ## 5. Submit and read your own writes A client submits commands and asks the read model. A command names the aggregate it goes to by domain, aggregate type and ID: `shop`, `cart`, `c1`. The comments show the output; the order ID will differ. submit.ts: ```ts import { Kernable } from "@kernable/sdk"; const kernable = new Kernable({ key: "app.key" }); const cartC1 = { domain: "shop", aggregateType: "cart", aggregateId: "c1" }; await kernable.submit({ ...cartC1, type: "AddItem", data: { sku: "apple", qty: 2 } }); // submit returned only after every listener processed the event: no polling needed. let cart = await kernable.query({ domain: "shop", type: "GetCart", data: { id: "c1" } }); console.log(cart); // { items: 2 } await kernable.submit({ ...cartC1, type: "CheckOut" }); cart = await kernable.query({ domain: "shop", type: "GetCart", data: { id: "c1" } }); console.log(cart); // { items: 2, order: '0b9c5e1a-6f3d-8a42-9c1e-5d7f2b8e4a10' } const outcome = await kernable.submit({ ...cartC1, type: "AddItem", data: { sku: "pear", qty: 1 } }); console.log(outcome); // { committed: false, code: 'rejected', message: 'Cart is checked out' } ``` submit.py: ```python from kernable import Kernable k = Kernable(key="app.key") cart_c1 = dict(domain="shop", aggregate_type="cart", aggregate_id="c1") k.submit(**cart_c1, type="AddItem", data={"sku": "apple", "qty": 2}) # submit returned only after every listener processed the event: no polling needed. cart = k.query(domain="shop", type="GetCart", data={"id": "c1"}) print(cart) # {'items': 2} k.submit(**cart_c1, type="CheckOut") cart = k.query(domain="shop", type="GetCart", data={"id": "c1"}) print(cart) # {'items': 2, 'order': '0b9c5e1a-6f3d-8a42-9c1e-5d7f2b8e4a10'} outcome = k.submit(**cart_c1, type="AddItem", data={"sku": "pear", "qty": 1}) print(outcome) # Outcome(committed=False, events=[], code='rejected', message='Cart is checked out', key='', requirement=None) ``` src/bin/submit.rs: ```rust use kernable::{BoxError, Kernable, Query, Submit}; use serde_json::{Value, json}; #[tokio::main] async fn main() -> Result<(), BoxError> { let kernable = Kernable::new("app.key")?; let get_cart_c1 = Query::new("shop", "GetCart").data(json!({ "id": "c1" })); let add_apples = Submit::new("shop", "cart", "c1", "AddItem").data(json!({ "sku": "apple", "qty": 2 })); kernable.submit(add_apples).await?; // submit returned only after every listener processed the event: no polling needed. let cart: Value = kernable.query(get_cart_c1.clone()).await?; println!("{cart}"); // {"items":2} kernable .submit(Submit::new("shop", "cart", "c1", "CheckOut")) .await?; let cart: Value = kernable.query(get_cart_c1).await?; println!("{cart}"); // {"items":2,"order":"0b9c5e1a-6f3d-8a42-9c1e-5d7f2b8e4a10"} let add_pears = Submit::new("shop", "cart", "c1", "AddItem").data(json!({ "sku": "pear", "qty": 1 })); let outcome = kernable.submit(add_pears).await?; println!("{outcome:?}"); // Rejected(Rejected { code: "rejected", message: "Cart is checked out", key: None, requirement: None }) Ok(()) } ``` In a fourth terminal, in `kernable-demo`, save the file and run it: ```sh [TypeScript] node submit.ts ``` ```sh [Python] py/bin/python submit.py # Windows: py\Scripts\python submit.py ``` ```sh [Rust] cargo run --bin submit # saved as src/bin/submit.rs ``` Now look at it in the UI, http://127.0.0.1:7234: - **Overview:** *Healthy*, and the commands and events so far. - **Explore › Domains:** the `shop` domain with the cart, its commands and the events it writes (`ItemAdded`, `CheckedOut`), read by the carts read model, which answers `GetCart`; green dots show both connected and up to date. - **Operations:** the worker and the read model, each with its key, since when, and how far behind. - **Search** `c1`: the cart and its events in order, each with the command that caused it and who sent it. ## 6. One app The worker and the read model ran as two processes. When they belong together, run them as one **app**: it runs everything defined on its domains and tells the kernel what the service consists of, so the kernel knows what to wait for (see [Remove a listener, middleware or aggregate](../how-to/remove-listener)). In one file with the aggregate and the read model, replace both run blocks with: ```ts [TypeScript] await new Kernable({ key: "app.key" }) .app({ service: "shop", version: "1.0.0", domains: [shopDomain] }) .run(); ``` ```python [Python] Kernable(key="app.key").app(service="shop", version="1.0.0", domains=[shop_domain]).run() ``` ```rust [Rust] use kernable::App; let kernable = Kernable::new("app.key")?; let app = kernable.app(App::new().service("shop").version("1.0.0").domain(&shop_domain))?; app.run().await?; ``` It prints one overview once everything is connected: ```text kernable · shop 1.0.0 kernel http://127.0.0.1:7233 · key app (69212a19f36f…) membership applied: 1 worker, 1 listener worker cart runs AddItem, CheckOut listener carts from 0 · reads ItemAdded, CheckedOut answers GetCart ready in 12 ms ``` ## 7. The reference application `examples/shop` is a two-domain shop. The `identity` domain is written in Python, the `orders` domain and the HTTP API in TypeScript. A reactor confirms orders using the other domain's read model. Its acceptance tests run everything over TCP with mutual TLS and kill each process once: ```sh make check # all tests, including examples/shop/shop.test.ts ``` ## Keys and grants Every process signs what it sends with its own Ed25519 key, so the kernel and `kernable verify` know who sent every command and event. A signature proves who sent a message, but anyone can make a key. So the kernel trusts one **root key**, and the root key **grants** each key what it may do: which commands it may submit or run, which events it may read, which queries it may ask or answer. `kernable start --dev` made a root key, `app.key` and a grant that lets `app` do everything; the SDKs send `app.grant` with every request signed by `app.key`. In production, every service gets its own key and only the grant it needs, and the root key stays offline: ```sh kernable keys new root && kernable keys new kernel && kernable keys new orders kernable grant root.key orders.pub 'run=shop.*' 'listen=shop.*' # writes orders.grant kernable start --root root.pub ``` [Part 8 of the shop tutorial](./shop/8-operate) sets this up for a whole application; [Trust](../concepts/trust) explains the model. ## Where to go next - The [reference](../reference/) covers every [kernel flag](../reference/configuration), the [CLI](../reference/cli) (also `kernable certs new` for mutual TLS), the SDKs, [metrics](../reference/metrics), the [protocol](../reference/protocol) and the exact [signed byte layouts](../reference/signing), with test vectors to write an SDK for another language. - Rules worth knowing: - **Command handlers must be deterministic.** Use `ctx.now()` and `ctx.id()`. - **Reactors submit with a fixed command ID**, so a redelivered event is harmless. Their submits settle on the listeners before them (the read models), not on themselves: a listener handles one event at a time and never waits for itself. The kernel refuses at once what would wait for the sending listener, such as a query to its own read model. - **A domain only writes its own events.** It reacts to other domains through listeners. - **Personal data and secrets go to the vault**, never into the signed log: mark the fields on the domain (`personal`, `secret`) and erase a person with an erasing event (`erases`); see `EventSchema` and `CommandSchema` in the [TypeScript SDK](../reference/sdk-typescript). - **Events are never changed.** To change an event's payload, raise its version and add an upcast from the old one: `shopDomain.event("ItemAdded", 2).upcast(1, (d) => ({ ...d, count: d.qty }))`. Listeners in other processes and languages need the same definition (`use(shopDomain)`); a listener that meets an unknown version stops until it is updated. --- # For LLMs and coding agents Starting a project on kernable with an AI assistant or a coding agent? Give it kernable's rules before it writes code. Three files exist for that: | File | What it is | Use it | |---|---|---| | kernable.AGENTS.md | a compact guide written as instructions: the mental model, project layout, every pattern with code, the rules, the typical errors and how to verify | copy it into your project as `AGENTS.md`, `CLAUDE.md` or your tool's context file | | [`/llms.txt`](/llms.txt) | a short index of this documentation, following the [llms.txt](https://llmstxt.org) convention | tools that look for it find it at the site's root | | [`/llms-full.txt`](/llms-full.txt) | the whole documentation in one Markdown file, with all code examples inlined | give it to an assistant that should know everything, or that answers questions about kernable | ## Start a project with an agent 1. Copy `kernable.AGENTS.md` into the new project, as `AGENTS.md` (or `CLAUDE.md`): ```sh curl -fsSL https://kernable.com/docs/kernable.AGENTS.md -o AGENTS.md ``` 2. Start a kernel for development in its own terminal: `kernable start --dev`. Its UI at http://127.0.0.1:7234 shows what the agent built. 3. Describe your domain to the agent in business terms: what is decided (aggregates, commands, events), what is read (queries), what happens as a consequence (reactors), who may do what (middleware), which data is personal. 4. Ask it to verify its work the way the file says: unit tests with `fold`, a run against the kernel, the UI. The file is written for TypeScript, with a table of the Python and Rust spellings; the tutorials show all three in full. ## The guide ```md # kernable: context for coding agents Put this file into a project that uses kernable (as `AGENTS.md`, `CLAUDE.md` or your tool's context file). It tells you, the agent, how to build on kernable correctly. Full documentation: https://kernable.com/docs/ — everything in one file: https://kernable.com/docs/llms-full.txt **Status:** kernable is in private preview: not released yet, the install scripts and SDK packages are not public, and the API may change before 1.0. Don't tell the user that a package or install command works unless they have early access. ## 1. Mental model - kernable is a **kernel** (one Rust process, `kernable start`) plus **your processes** in any language (TypeScript, Python, Rust SDKs). Your processes hold the business logic; the kernel checks, orders, stores and delivers. - Business logic is **event-sourced**: a **domain** (`shop`) holds **aggregates** (`cart`). A **command** (`AddItem`) goes to one aggregate; its handler looks at the aggregate's state and returns **events** (`ItemAdded`) or rejects. `apply` folds events into state. Events are stored forever and never changed. - **Read models** build views from events and answer **queries** (`GetCart`). **Reactors** react to events with side effects and follow-up commands. A **middleware** is a listener that checks commands and queries. - Every command and query goes **through the kernel**, signed with the process's key and allowed by its grant. There are no shortcuts, also not inside one process. - A `submit` **settles** by default: it returns once every listener has processed the events, so the next query sees them. No polling, no eventual consistency for the reader. - One command per aggregate at a time; different aggregates in parallel. Listeners get events in one global order, read models (order 0) before reactors (order 10). ## 2. Set up a project ```sh mkdir shop && cd shop kernable start --dev # in its own terminal: creates app.key, app.grant, vault.key; UI at http://127.0.0.1:7234 npm init -y && npm pkg set type=module && npm install @kernable/sdk # Node >= 22.18 runs .ts directly ``` Layout: ``` shop/ domain.ts export const shopDomain = domain("shop"); event versions; vault declarations cart.ts the cart aggregate, defined on shopDomain carts.ts a read model on shopDomain mails.ts a reactor on shopDomain app.ts runs everything: kernable.app({ service, version, domains }) ``` Every file defines its part on the domain object it imports. The entry point: ```ts import { Kernable } from "@kernable/sdk"; import { shopDomain } from "./shop/domain.ts"; import "./shop/cart.ts"; import "./shop/carts.ts"; await new Kernable().app({ service: "shop", version: "1.0.0", domains: [shopDomain] }).run(); ``` `new Kernable()` reads `KERNABLE_URL` (default `http://127.0.0.1:7233`), `KERNABLE_KEY` (default `app.key`, with `app.grant` next to it) and `KERNABLE_SERVICE`. `app()` applies the service's membership (what the kernel waits for) and runs every aggregate, read model and reactor of its domains. ## 3. Patterns ### Aggregate ```ts import { Rejected, domain } from "@kernable/sdk"; export const shopDomain = domain("shop"); type Cart = { items: number; total: number; checkedOut: boolean }; export const cartAggregate = shopDomain.aggregate("cart", { items: 0, total: 0, checkedOut: false }, { snapshotEvery: 100 }) .apply("ItemAdded", (s, d: { qty: number; price: number }) => { s.items += d.qty; s.total += d.qty * d.price; }) .apply("CheckedOut", (s) => { s.checkedOut = true; }) .command("AddItem", (s, d: { sku: string; qty: number; price: number }) => { if (s.checkedOut) throw new Rejected("the cart is checked out", "checked_out"); if (!(d.qty > 0)) throw new Rejected("quantity must be positive", "invalid_quantity"); return [{ type: "ItemAdded", data: d }]; }) .command("CheckOut", (s, _d, cmd, ctx) => { if (s.items === 0) throw new Rejected("the cart is empty", "empty"); return [{ type: "CheckedOut", data: { orderId: ctx.id(), at: ctx.now().toISOString(), total: s.total } }]; }); ``` - `apply(type, (state, data, evt, ctx) => …)` changes `state` in place (it is a copy) or returns a new state. No clock, no IDs, no I/O. - `command(type, (state, data, cmd, ctx) => events)` returns events or throws `Rejected(message, code)`. Use `ctx.now()` and `ctx.id()`, never `Date.now()` or random IDs: a command may run again and must decide the same way. - The state must survive a JSON round trip (snapshots store it as JSON). ### Read model with a query ```ts const carts = new Map(); shopDomain.readModel("carts") .on("ItemAdded", (d: { qty: number }, evt, ctx) => { const cart = carts.get(evt.aggregateId) ?? { items: 0, owner: ctx.request.senderIdentity }; cart.items += d.qty; carts.set(evt.aggregateId, cart); }) .on("CheckedOut", (d: { orderId: string }, evt) => { carts.get(evt.aggregateId)!.order = d.orderId; }) .query("GetCart", (d: { id: string }) => { const cart = carts.get(d.id); if (!cart) throw new Rejected(`no cart ${d.id}`, "not_found"); return cart; }); ``` - A read model kept in memory rebuilds from the first event on every start (`start: 0`). To keep it across restarts: `readModel("carts", { store: "sqlite:carts.db" })` and keep state in `ctx.store` (`get`, `set`, `remove`, `all`); raise `schema` when you change how it stores data. - Handlers get events **at least once**: write them so that a repeated event changes nothing (overwrite, do not blindly add, or use a store). - Another domain's events are qualified: `.on("billing.InvoicePaid", …)`; add `.use(billingDomain)` to get its event versions. ### Reactor with a follow-up command ```ts billingDomain.reactor("invoicing").on("shop.CheckedOut", async (d: { orderId: string; total: number }, _evt, ctx) => { // Keeps the sender and the correlation; a rejection parks the event. await ctx.submit({ aggregateType: "invoice", aggregateId: d.orderId, type: "CreateInvoice", data: { orderId: d.orderId, total: d.total }, id: `invoice-${d.orderId}`, // fixed ID: a redelivered event changes nothing }); }); ``` - Reactors start at the newest event when they are new (they never react to history) and resume after their last acknowledgement. - Submit follow-ups with `ctx.submit` (it needs a **fixed command ID** derived from the event, and follows the event's sender), query with `ctx.query`. - A rejected follow-up throws: the reactor **parks** the event (after 5 failed attempts for other errors) and moves on; operators see it in `kernable listeners parked` and retry it. Catch the `Rejected` when a rejection is fine (`unique_taken` of an earlier attempt). A read model never parks. - Side effects outside kernable (mail, HTTP) happen at least once: make them idempotent. ### Middleware ```ts const known = new Set(); identityDomain.readModel("rbac") .on("CustomerRegistered", (_d, evt) => void known.add(evt.aggregateId)) .middleware({ types: ["shop.*"], order: 0 }, (check) => { const who = check.request.senderIdentity; if (!who) throw new Rejected("who are you?", "unauthenticated"); if (!known.has(who)) throw new Rejected(`${who} is no customer`, "unknown_customer"); }); ``` - Decide in memory from what the middleware learned from events; no database or network calls. - `when: "after"` runs after a request: it may refuse a query's answer, and observes a command's outcome. - A middleware fails closed: while it is away, what it checks is refused (`UNAVAILABLE`). ### Client: submit and query ```ts const kernable = new Kernable({ context: { senderIdentity: "alice" } }); const outcome = await kernable.submit({ domain: "shop", aggregateType: "cart", aggregateId: "c1", type: "AddItem", data: { sku: "apple", qty: 2, price: 3 } }); if (!outcome.committed) console.log(outcome.code, outcome.message); // a rejection is a result, not an exception const cart = await kernable.query({ domain: "shop", type: "GetCart", data: { id: "c1" } }); // already shows the item ``` - `submit` waits until settled (default); `wait: "committed"` returns once stored; `enqueue` returns only the ID. - `id` makes a submit idempotent: the same ID returns the original outcome and never runs twice. - `query` throws `Rejected` when the read model rejects. - `context` carries who the request is for (`senderIdentity`, `senderTenant`, …); middleware decides on it. `attributes` are business data only. ### Personal data and secrets ```ts shopDomain.command("Register").personal(["email", "name"], (_d, id) => id).secret(["password"]); shopDomain.event("CustomerRegistered", 1).personal(["email", "name"], (_d, id) => id); shopDomain.event("CustomerErased", 1).erases((_d, id) => id); // a client that submits these commands: new Kernable().use(shopDomain) ``` Personal fields go to the vault (the log keeps references), secret fields only until their command completes, and an erasing event deletes a person's values everywhere in its tenant. Only top-level fields. Files and cache values are not erased with the person. ### Changing an event's format ```ts shopDomain.event("ItemAdded", 2).upcast(1, (d: { sku: string; quantity: number }) => ({ sku: d.sku, qty: d.quantity })); ``` Never change or delete stored events. Raise the version and add an upcast from each older one; deploy readers before writers. A new optional field that every reader can ignore needs no new version. ## 4. Python and Rust Same concepts, same order of arguments. The tested examples in all three languages: https://kernable.com/docs/tutorial/getting-started | | TypeScript | Python | Rust | |---|---|---|---| | domain | `domain("shop")` | `domain("shop")` | `domain("shop")` | | aggregate | `.aggregate("cart", initial)` | `.aggregate("cart", initial)` | `.aggregate("cart", Cart::default())` (state: `Clone + Default + Serialize + Deserialize`) | | handlers | `.apply(type, fn)`, `.command(type, fn)` | decorators: `@cart.apply("ItemAdded")`, `@cart.command("AddItem")` | `.apply(type, \|state, data: T, evt, ctx\| …)`, `.command(type, \|state, data: T, cmd, ctx\| Ok(vec![NewEvent::new(type, data)]))` | | reject | `throw new Rejected(msg, code)` | `raise Rejected(msg, code)` | `Err(Rejected::new(msg).code(code))` | | read model | `.readModel("carts")` | `.read_model("carts")` | `.read_model("carts")` | | handlers on it | `.on(type, (data, evt, ctx) => …)` | `@carts.on("ItemAdded")` with `(data, evt)` | `.on(type, \|data: T, evt, ctx\| async { Ok(()) })` | | query | `.query(type, (data) => …)` | `@carts.query("GetCart")` | `.query(type, \|data: T, qry, ctx\| async { Ok(answer) })` | | middleware | `.middleware({ types, order }, fn)` | `@rbac.middleware(["shop.*"], order=0)` | `.middleware(MiddlewareOptions::new(&["shop.*"]), \|check\| async { … })` | | client | `new Kernable()` | `Kernable()` | `Kernable::new("app.key")?` | | submit | `kernable.submit({ … })` | `k.submit(domain=, aggregate_type=, aggregate_id=, type=, data=, id=, context=)` | `kernable.submit(Submit::new(domain, aggregate_type, id, type).data(json!(…)).id(…).context(…)).await?` | | query | `kernable.query({ domain, type, data })` | `k.query(domain=, type=, data=)` | `kernable.query(Query::new(domain, type).data(json!(…))).await?` | | app | `kernable.app({ service, version, domains }).run()` | `k.app(service=, version=, domains=[…]).run()` (blocks) | `kernable.app(App::new().service(…).version(…).domain(&d))?.run().await?` | | field names | `aggregateId`, `senderIdentity` | `aggregate_id`, `sender_identity` | `aggregate_id`, `sender_identity` | Python runs handlers in threads and `run()` blocks; Rust is async (Tokio) and keeps handler state in `Arc>`. ## 5. Rules: always and never Always: - Decide only from the aggregate's own state and the command's data; use `ctx.now()` and `ctx.id()`. - Name events in the past tense (`ItemAdded`), commands in the imperative (`AddItem`), queries as questions (`GetCart`); domains and aggregates lowercase. - Keep aggregates small: one is the unit of consistency; rules across aggregates go through a reactor and follow-up commands. - Declare values that must be unique across aggregates as `unique` keys, and conditions on another aggregate as a `fact` it holds and a command `requires`: the kernel checks both at commit. Never build a registry of claims or check a read model before a submit for this. - A requirement names the other aggregate by its ID, or by a unique key and value (`{ aggregateType: "vehicle", key: "isin", value, fact: "open" }`, normalised as the key is), so no lookup of the ID is needed first; `absent: true` requires the fact not to be held (an aggregate that does not exist holds none). Omit `domain` for the command's own. - Give follow-up and retried commands a fixed ID; submit them from reactors with `ctx.submit`. - Mark personal data `personal` and passwords or tokens `secret`. - Run everything through `app()` (or apply the membership once per deploy) so the kernel knows what to wait for. Never: - Never call the clock, random IDs, the network, a database or a query from a command handler or `apply`. - Never write another domain's events: react to them through a listener instead. - Never read another process's state directly: ask a query through the kernel. - Never change, delete or "fix" stored events: add new ones, or raise the version with an upcast. - Never put personal data into IDs, event types or the request context: they are not erased. - Never let a listener ask its own read model, or a middleware send what it checks itself: the kernel refuses it. ## 6. When something goes wrong | You see | Cause | Do | |---|---|---| | `NOT_FOUND` on submit | no worker registered the aggregate or command type, and no membership lists it (often a typo) | start the worker or app first; check domain, aggregate type and command type | | `UNAVAILABLE` | a middleware that checks the request is away, or no read model answers the query | start it; the UI (http://127.0.0.1:7234) shows what is missing | | `PERMISSION_DENIED` | the key's grant does not allow it | with `--dev`, app.key may do everything; in production, grant it (`kernable grant`) | | `NotSettled` | a listener did not process the events in time (default 5 s); the command **is** committed | find the listener that is behind (UI, `/ready`); fix its handler | | rejected with `handler_error` | a command handler threw something other than `Rejected` | fix the bug; resubmit with a new command ID | | rejected with `version_mismatch` | `expectedVersion` in the context did not match | reload and retry | | a listener restarts in a loop | its handler throws on one event | its log shows the error and `connected again at position N`; it is stuck at the event after N | | a process stops at an event | the event has a newer schema than the process knows | deploy the reader with the new event version | ## 7. Verify your work 1. **Unit-test the domain without a kernel**: fold events and call handlers. ```ts const state = cartAggregate.fold([{ type: "ItemAdded", schema: 1, data: { sku: "a", qty: 2, price: 3 } }]); // then call the command handler on that state and assert the events or the rejection ``` 2. **Run against a real kernel**: `kernable start --dev --store memory: --listen 127.0.0.1:17233` in a temporary directory, point `KERNABLE_URL` and `KERNABLE_KEY` at it, start the app, `await app.ready`, then submit and query. Settled submits mean the next query must see the change. 3. **Look**: the UI at http://127.0.0.1:7234 says whether anything needs attention (Overview), shows the services and their lag (Operations), the domains with their commands, events and readers, the log and every aggregate's history with its commands (Explore), and the business processes with their cases (Processes). 4. **Check the log**: `kernable verify kernable.db kernel.pub` recomputes the hash chain and every signature. ## 8. Where to read more - Concepts: https://kernable.com/docs/concepts/ (architecture, event sourcing, domains, ordering, trust, queries and middleware, data, failures) - How-to guides: https://kernable.com/docs/how-to/ - Reference (SDKs, CLI, flags, protocol, errors): https://kernable.com/docs/reference/ - Everything in one file: https://kernable.com/docs/llms-full.txt ``` --- # Build a shop In ten parts you build a small shop on kernable: customers register and place orders, a service confirms every order by email, customers may only touch their own data, and a customer can have their personal data erased. Part 8 runs it as it would in production; part 9 adds accounts, sign-in and an HTTP API with kernable's kit, and part 10 a business process that ships paid orders and cancels unpaid ones. Every part shows the code in **TypeScript, Python and Rust**; pick one with the tabs, or mix them: part 5 is about exactly that. | Part | You add | You learn | |---|---|---| | [1. Run a kernel](./1-kernel) | the kernel, the SDK | what the kernel does and what your processes do | | [2. Aggregates and commands](./2-commands) | `customer`, `order`, the worker | commands, events, rejections, idempotency | | [3. Read models and queries](./3-queries) | `customers`, `orders` | queries through the kernel, reading your own writes | | [4. A reactor](./4-reactor) | `confirmations` | listener order, follow-up commands | | [5. A second language](./5-languages) | nothing new | mixing languages in one application | | [6. Authorization](./6-authorization) | `guard` | grants and middleware, the request context | | [7. Personal data](./7-personal-data) | vault declarations, erasure | the vault, secrets, erasing a person | | [8. Operate](./8-operate) | a production setup | a key per service, an offline root key, rotation and revocation, mutual TLS, metrics, persistent read models, deploys | | [9. Accounts and sign-in](./9-sign-in) | `api`, the kit, a rule of the shop | accounts and sessions, tenants, groups, invitation links, a query's target, `Explain`, rules on data next to the kit | | [10. A business process](./10-process) | `fulfilment` | processes run by the kernel: cases, waits with a timeout, incidents, the UI's diagram and statistics | From part 2 on, the snippets show what a part adds to a file, without its imports. The complete files, as they are at the end, are in the repository under `docs/examples/shop/ts`, `docs/examples/shop/py` and `docs/examples/shop/rs`; kernable's tests run them for every part, in every language. You need Node.js ≥ 22.18 for TypeScript, Python ≥ 3.10 for Python or Rust (stable) for Rust, and about an hour. --- # 1. Run a kernel kernable splits an application in two. The **kernel** stores commands and events, signs and checks every message, runs commands one at a time per aggregate and delivers events in order. **Your processes** hold the business logic: they decide what a command means and what to do with an event. They connect to the kernel over gRPC, in any language. ## Install kernable is in private preview: the install scripts and the SDK packages (`@kernable/sdk` on npm, `kernable` on PyPI and crates.io) are not public yet, so the commands below do not work for everyone yet. [Request early access](https://kernable.com/#access) to get the binaries and the SDKs. Install the kernel as in [Getting started](../getting-started#_1-install-the-kernel): ```sh [macOS, Linux] curl -fsSL https://kernable.com/install.sh | sh ``` ```powershell [Windows] irm https://kernable.com/install.ps1 | iex ``` ## Start the kernel ```sh mkdir -p ~/shop && cd ~/shop ``` ```sh kernable start --dev ``` It listens on `http://127.0.0.1:7233`, for this machine only, and keeps the event log in `kernable.db` in the current directory. `--dev` creates the keys the shop uses, among them `app.key`, which may do everything here, and turns on the vault for personal data (part 7). Part 8 shows how keys and grants work in production. Leave the kernel running and continue in a second terminal. ## Set up the project Make the shop a project in the language you write it in, with the SDK: ```sh [TypeScript] cd ~/shop npm init -y >/dev/null && npm pkg set type=module npm install @kernable/sdk ``` ```sh [Python] cd ~/shop python3 -m venv .venv && .venv/bin/pip install -q kernable ``` ```sh [Rust] cd ~/shop cargo init -q --lib --name shop cargo add -q kernable serde_json scrypt cargo add -q serde --features derive cargo add -q tokio --features macros,rt-multi-thread ``` In Rust, `domain.rs` (part 2) lives in `src/` and `src/lib.rs` shares it; every process and script is a binary in `src/bin/` (run with `cargo run -q --bin worker`): ```rs //! The shop's domain. Every binary in `src/bin` is one process or one part's script. mod domain; pub use domain::{Shop, shop}; ``` ## Connect Every process of the shop connects with a client and no options: ```ts [TypeScript] const kernable = new Kernable(); ``` ```python [Python] k = Kernable() ``` ```rust [Rust] let kernable = Kernable::new(Options::default())?; ``` The SDK takes the kernel's address from `KERNABLE_URL`, the key from `KERNABLE_KEY` and the TLS files from `KERNABLE_TLS_CA`, `KERNABLE_TLS_CERT` and `KERNABLE_TLS_KEY`. Without them, it connects to the kernel you just started with `app.key` from the current directory, and the grant `--dev` put next to it. So the processes run unchanged in part 8, where each gets its own key and certificate. Options in the code take precedence over the environment. Workers and listeners print a line to stderr when they connect, with the kernel's address, the key and what they run. `KERNABLE_LOG=debug` adds every command, event and query they handle, `KERNABLE_LOG=warn` keeps only problems, `off` silences them. Next: [2. Aggregates and commands](./2-commands). --- # 2. Aggregates and commands An **aggregate** is one thing your business makes decisions about: a customer, an order. It has a **state**, built from its **events**, and **commands** that ask it to change. A command handler looks at the state and either returns new events or rejects the command. Nothing else changes state: the events are the truth, and the kernel stores them forever. Aggregates belong to a **domain**, here `shop`: one area of the business, and in code one object that holds everything of it, its aggregates, read models and reactors, its event versions and which fields go to the vault. Types are unique within a domain and named `shop.Register` elsewhere. A domain writes only its own events and learns about other domains through listeners; [Domains](../../concepts/domains) explains the rules and a code layout for when it grows. ## The domain Start `domain.ts` (or `domain.py`, `src/domain.rs`) with the domain and the customer: domain.ts: ```ts export const shopDomain = domain("shop"); ``` domain.py: ```py shop_domain = domain("shop") ``` domain.rs: ```rs let shop_domain = domain("shop"); ``` domain.ts: ```ts type Customer = { registered: boolean; passwordHash?: string; salt?: string }; export const customerAggregate = shopDomain.aggregate("customer", { registered: false }) .apply("CustomerRegistered", (s) => { s.registered = true; }) .command("Register", (s, data: { email: string; name: string }) => { if (s.registered) throw new Rejected("Customer already registered", "exists"); if (!/^[^@\s]+@[^@\s]+$/.test(data.email ?? "")) throw new Rejected("Invalid email", "invalid"); return [{ type: "CustomerRegistered", data: { email: data.email, name: data.name } }]; }); ``` domain.py: ```py customer_aggregate = shop_domain.aggregate("customer", {"registered": False}) @customer_aggregate.apply("CustomerRegistered") def _(state, data): state["registered"] = True @customer_aggregate.command("Register") def _(state, data): if state["registered"]: raise Rejected("Customer already registered", "exists") if not re.match(r"^[^@\s]+@[^@\s]+$", data.get("email") or ""): raise Rejected("Invalid email", "invalid") return [{"type": "CustomerRegistered", "data": {"email": data["email"], "name": data["name"]}}] ``` domain.rs: ```rs #[derive(Clone, Default, Serialize, Deserialize)] pub struct Customer { registered: bool, password_hash: Option, salt: Option, } #[derive(Deserialize)] struct Register { email: Option, name: Option, } fn customer(shop_domain: &Domain) -> Aggregate { let customer_aggregate = shop_domain.aggregate("customer", Customer::default()); customer_aggregate.apply("CustomerRegistered", |s, _: Value, _evt, _ctx| { s.registered = true }); customer_aggregate.command("Register", |s, data: Register, _cmd, _ctx| { if s.registered { return Err(Rejected::new("Customer already registered").code("exists")); } let email = data.email.unwrap_or_default(); let valid = email.split_once('@').is_some_and(|(user, host)| { !user.is_empty() && !host.is_empty() && !email.contains(char::is_whitespace) && !host.contains('@') }); if !valid { return Err(Rejected::new("Invalid email").code("invalid")); } Ok(vec![NewEvent::new( "CustomerRegistered", json!({ "email": email, "name": data.name }), )]) }); customer_aggregate } ``` `apply` turns an event's data into the next state; `command` decides. `Rejected` carries a message and a code (`exists`, `invalid`) your callers can act on. Add the order: domain.ts: ```ts type Order = { status: "none" | "draft" | "submitted" | "confirmed" | "paid" | "shipped" | "cancelled"; items: number }; export const orderAggregate = shopDomain.aggregate("order", { status: "none", items: 0 }) .apply("OrderPlaced", () => ({ status: "draft", items: 0 })) .apply("ItemAdded", (s, data: { qty: number }) => { s.items += data.qty; }) .apply("OrderSubmitted", (s) => { s.status = "submitted"; }) .apply("OrderConfirmed", (s) => { s.status = "confirmed"; }) .command("Place", (s, data: { customerId: string }) => { if (s.status !== "none") throw new Rejected("Order already exists", "exists"); return [{ type: "OrderPlaced", data: { customerId: data.customerId } }]; }) .command("AddItem", (s, data: { sku: string; qty: number }) => { if (s.status !== "draft") throw new Rejected(`Cannot add items to a ${s.status} order`, "invalid_state"); if (!(data.qty > 0)) throw new Rejected("Quantity must be positive", "invalid"); return [{ type: "ItemAdded", data: { sku: data.sku, qty: data.qty } }]; }) .command("Submit", (s) => { if (s.status !== "draft" || s.items === 0) throw new Rejected(`Cannot submit a ${s.status} order with ${s.items} items`, "invalid_state"); return [{ type: "OrderSubmitted" }]; }); ``` domain.py: ```py order_aggregate = shop_domain.aggregate("order", {"status": "none", "items": 0}) @order_aggregate.apply("OrderPlaced") def _(state, data): return {"status": "draft", "items": 0} @order_aggregate.apply("ItemAdded") def _(state, data): state["items"] += data["qty"] @order_aggregate.apply("OrderSubmitted") def _(state, data): state["status"] = "submitted" @order_aggregate.apply("OrderConfirmed") def _(state, data): state["status"] = "confirmed" @order_aggregate.command("Place") def _(state, data): if state["status"] != "none": raise Rejected("Order already exists", "exists") return [{"type": "OrderPlaced", "data": {"customerId": data["customerId"]}}] @order_aggregate.command("AddItem") def _(state, data): if state["status"] != "draft": raise Rejected(f"Cannot add items to a {state['status']} order", "invalid_state") if not data.get("qty", 0) > 0: raise Rejected("Quantity must be positive", "invalid") return [{"type": "ItemAdded", "data": {"sku": data["sku"], "qty": data["qty"]}}] @order_aggregate.command("Submit") def _(state, data): if state["status"] != "draft" or state["items"] == 0: raise Rejected(f"Cannot submit a {state['status']} order with {state['items']} items", "invalid_state") return [{"type": "OrderSubmitted"}] ``` domain.rs: ```rs #[derive(Clone, Serialize, Deserialize)] pub struct Order { /// none, draft, submitted, confirmed, paid, shipped or cancelled. status: String, items: i64, } #[derive(Serialize, Deserialize)] #[serde(rename_all = "camelCase")] struct Place { customer_id: String, } #[derive(Serialize, Deserialize)] struct AddItem { sku: String, qty: Option, } fn order(shop_domain: &Domain) -> Aggregate { let order_aggregate = shop_domain.aggregate( "order", Order { status: "none".into(), items: 0, }, ); order_aggregate.apply("OrderPlaced", |s, _: Value, _evt, _ctx| { *s = Order { status: "draft".into(), items: 0, } }); order_aggregate.apply("ItemAdded", |s, data: AddItem, _evt, _ctx| { s.items += data.qty.unwrap_or(0) }); order_aggregate.apply("OrderSubmitted", |s, _: Value, _evt, _ctx| { s.status = "submitted".into() }); order_aggregate.apply("OrderConfirmed", |s, _: Value, _evt, _ctx| { s.status = "confirmed".into() }); order_aggregate.command("Place", |s, data: Place, _cmd, _ctx| { if s.status != "none" { return Err(Rejected::new("Order already exists").code("exists")); } Ok(vec![NewEvent::new("OrderPlaced", data)]) }); order_aggregate.command("AddItem", |s, data: AddItem, _cmd, _ctx| { if s.status != "draft" { let message = format!("Cannot add items to a {} order", s.status); return Err(Rejected::new(message).code("invalid_state")); } if data.qty.is_none_or(|qty| qty <= 0) { return Err(Rejected::new("Quantity must be positive").code("invalid")); } Ok(vec![NewEvent::new("ItemAdded", data)]) }); order_aggregate.command("Submit", |s, _: Value, _cmd, _ctx| { if s.status != "draft" || s.items == 0 { let message = format!("Cannot submit a {} order with {} items", s.status, s.items); return Err(Rejected::new(message).code("invalid_state")); } Ok(vec![NewEvent::new("OrderSubmitted", ())]) }); order_aggregate } ``` Handlers must be **deterministic** and free of side effects: the kernel may run a command again (after a crash, or when another command changed the aggregate in between), and it must decide the same way. ## The worker A **worker** runs the commands of its aggregates. It connects, tells the kernel which command types it handles, and waits: worker.ts: ```ts const kernable = new Kernable(); await kernable.worker(customerAggregate, orderAggregate).run(); ``` worker.py: ```py k = Kernable() k.worker(customer_aggregate, order_aggregate).run() ``` worker.rs: ```rs let kernable = Kernable::new(Options::default())?; kernable .worker(&shop.customer_aggregate) .aggregate(&shop.order_aggregate) .run() .await; ``` ```sh [TypeScript] node worker.ts ``` ```sh [Python] .venv/bin/python worker.py ``` ```sh [Rust] cargo run -q --bin worker ``` You can start more workers; the kernel spreads the commands over them and never runs two commands of one aggregate at a time. ## Send commands part2.ts: ```ts const send = (aggregateType: string, aggregateId: string, type: string, data?: object, id?: string) => kernable.submit({ domain: "shop", aggregateType, aggregateId, type, data, id }); show("register", await send("customer", "c1", "Register", { email: "ann@example.com", name: "Ann" })); show("register again", await send("customer", "c1", "Register", { email: "ann@example.com", name: "Ann" })); show("place", await send("order", "o1", "Place", { customerId: "c1" })); show("add apples", await send("order", "o1", "AddItem", { sku: "apple", qty: 2 }, "add-1")); show("same command again", await send("order", "o1", "AddItem", { sku: "apple", qty: 2 }, "add-1")); show("submit", await send("order", "o1", "Submit")); show("add after submit", await send("order", "o1", "AddItem", { sku: "pear", qty: 1 })); ``` part2.py: ```py def send(aggregate_type, aggregate_id, type, data=None, id=None): return k.submit("shop", aggregate_type, aggregate_id, type, data, id=id) show("register", send("customer", "c1", "Register", {"email": "ann@example.com", "name": "Ann"})) show("register again", send("customer", "c1", "Register", {"email": "ann@example.com", "name": "Ann"})) show("place", send("order", "o1", "Place", {"customerId": "c1"})) show("add apples", send("order", "o1", "AddItem", {"sku": "apple", "qty": 2}, "add-1")) show("same command again", send("order", "o1", "AddItem", {"sku": "apple", "qty": 2}, "add-1")) show("submit", send("order", "o1", "Submit")) show("add after submit", send("order", "o1", "AddItem", {"sku": "pear", "qty": 1})) ``` part2.rs: ```rs let send = |aggregate_type, aggregate_id, command_type, data: Value| { kernable.submit(Submit::new("shop", aggregate_type, aggregate_id, command_type).data(data)) }; let ann = json!({ "email": "ann@example.com", "name": "Ann" }); show( "register", send("customer", "c1", "Register", ann.clone()).await?, ); show( "register again", send("customer", "c1", "Register", ann).await?, ); show( "place", send("order", "o1", "Place", json!({ "customerId": "c1" })).await?, ); let apples = Submit::new("shop", "order", "o1", "AddItem") .data(json!({ "sku": "apple", "qty": 2 })) .id("add-1"); show("add apples", kernable.submit(apples.clone()).await?); show("same command again", kernable.submit(apples).await?); show("submit", send("order", "o1", "Submit", Value::Null).await?); show( "add after submit", send("order", "o1", "AddItem", json!({ "sku": "pear", "qty": 1 })).await?, ); ``` TypeScript: ```text register: committed, version 1 register again: exists (Customer already registered) place: committed, version 1 add apples: committed, version 2 same command again: committed, version 2 submit: committed, version 3 add after submit: invalid_state (Cannot add items to a submitted order) ``` Python: ```text register: committed, version 1 register again: exists (Customer already registered) place: committed, version 1 add apples: committed, version 2 same command again: committed, version 2 submit: committed, version 3 add after submit: invalid_state (Cannot add items to a submitted order) ``` Rust: ```text register: committed, version 1 register again: exists (Customer already registered) place: committed, version 1 add apples: committed, version 2 same command again: committed, version 2 submit: committed, version 3 add after submit: invalid_state (Cannot add items to a submitted order) ``` What happened: - Each committed command reports the aggregate's new **version**: one per event. - `register again` and `add after submit` were **rejected** by the handlers. A rejected command is stored as rejected, like a committed one. - `same command again` used the same command **ID** (`add-1`) and got the original outcome back: the item was added once. Give a command a fixed ID when you might send it twice, for example when retrying after a timeout. Next: [3. Read models and queries](./3-queries). --- # 3. Read models and queries Aggregates decide; they do not answer questions like "what is in order o1?". That is the job of **read models**: listeners that receive events in order and keep a view of them, shaped for the questions you ask. Read models are defined on the domain, like aggregates, and run through the connection with `listener`, like an aggregate with `worker`. A read model's state is private. Everything else reads it through **queries**, which go through the kernel like commands do: they are signed, allowed by the sender's grant, and checked by middleware (part 6). ## The read models Create `read-models.ts` (or `read_models.py`, `src/bin/read_models.rs`) with the customers: read-models.ts: ```ts type CustomerRow = { id: string; email: string; name: string }; const customers = new Map(); const customersReadmodel = shopDomain.readModel("customers") .on("CustomerRegistered", (data: { email: string; name: string }, evt) => { customers.set(evt.aggregateId, { id: evt.aggregateId, email: data.email, name: data.name }); }) .query("GetCustomer", (data: { id: string }) => { const row = customers.get(data.id); if (!row) throw new Rejected(`No customer ${data.id}`, "not_found"); return row; }); ``` read_models.py: ```py customers: dict[str, dict] = {} customers_readmodel = shop_domain.read_model("customers") @customers_readmodel.on("CustomerRegistered") def _(data, evt): customers[evt.aggregate_id] = {"id": evt.aggregate_id, "email": data["email"], "name": data["name"]} @customers_readmodel.query("GetCustomer") def _(data): if data["id"] not in customers: raise Rejected(f"No customer {data['id']}", "not_found") return customers[data["id"]] ``` read_models.rs: ```rs #[derive(Clone, Serialize, Deserialize)] struct CustomerRow { id: String, email: String, name: String, } #[derive(Deserialize)] struct Registered { email: String, name: String, } let customers = Arc::new(Mutex::new(HashMap::::new())); let rows = Arc::clone(&customers); let customers_readmodel = shop_domain.read_model("customers").on( "CustomerRegistered", move |data: Registered, evt, _ctx| { let Registered { email, name } = data; let id = evt.aggregate_id; rows.lock() .unwrap() .insert(id.clone(), CustomerRow { id, email, name }); async { Ok(()) } }, ); let rows = Arc::clone(&customers); customers_readmodel.query("GetCustomer", move |data: Id, _qry, _ctx| { let row = rows.lock().unwrap().get(&data.id).cloned(); async move { Ok(row.ok_or_else(|| { Rejected::new(format!("No customer {}", data.id)).code("not_found") })?) } }); ``` and the orders: read-models.ts: ```ts type OrderRow = { id: string; customerId: string; status: string; items: Record; confirmedTo?: string | null; cancelled?: string }; const orders = new Map(); const ordersReadmodel = shopDomain.readModel("orders") .on("OrderPlaced", (data: { customerId: string }, evt) => { orders.set(evt.aggregateId, { id: evt.aggregateId, customerId: data.customerId, status: "draft", items: {} }); }) .on("ItemAdded", (data: { sku: string; qty: number }, evt) => { const items = orders.get(evt.aggregateId)!.items; items[data.sku] = (items[data.sku] ?? 0) + data.qty; }) .on("OrderSubmitted", (_data, evt) => void (orders.get(evt.aggregateId)!.status = "submitted")) .query("GetOrder", (data: { id: string }) => { const row = orders.get(data.id); if (!row) throw new Rejected(`No order ${data.id}`, "not_found"); return row; }); ``` read_models.py: ```py orders: dict[str, dict] = {} orders_readmodel = shop_domain.read_model("orders") @orders_readmodel.on("OrderPlaced") def _(data, evt): orders[evt.aggregate_id] = {"id": evt.aggregate_id, "customerId": data["customerId"], "status": "draft", "items": {}} @orders_readmodel.on("ItemAdded") def _(data, evt): items = orders[evt.aggregate_id]["items"] items[data["sku"]] = items.get(data["sku"], 0) + data["qty"] @orders_readmodel.on("OrderSubmitted") def _(data, evt): orders[evt.aggregate_id]["status"] = "submitted" @orders_readmodel.query("GetOrder") def _(data): if data["id"] not in orders: raise Rejected(f"No order {data['id']}", "not_found") return orders[data["id"]] ``` read_models.rs: ```rs #[derive(Clone, Serialize)] #[serde(rename_all = "camelCase")] struct OrderRow { id: String, customer_id: String, status: String, items: serde_json::Map, #[serde(skip_serializing_if = "Option::is_none")] confirmed_to: Option>, #[serde(skip_serializing_if = "Option::is_none")] cancelled: Option, } let orders = Arc::new(Mutex::new(HashMap::::new())); let rows = Arc::clone(&orders); let orders_readmodel = shop_domain .read_model("orders") .on("OrderPlaced", move |data: Value, evt, _ctx| { let customer_id = data["customerId"].as_str().unwrap_or_default().to_owned(); let id = evt.aggregate_id; let row = OrderRow { id: id.clone(), customer_id, status: "draft".into(), items: Default::default(), confirmed_to: None, cancelled: None, }; rows.lock().unwrap().insert(id, row); async { Ok(()) } }); let rows = Arc::clone(&orders); orders_readmodel.on("ItemAdded", move |data: Value, evt, _ctx| { let (sku, qty) = ( data["sku"].as_str().unwrap_or_default(), data["qty"].as_i64().unwrap_or(0), ); let mut rows = rows.lock().unwrap(); let items = &mut rows.get_mut(&evt.aggregate_id).expect("placed").items; let count = items.get(sku).and_then(|n| n.as_i64()).unwrap_or(0) + qty; items.insert(sku.to_owned(), count.into()); async { Ok(()) } }); let rows = Arc::clone(&orders); orders_readmodel.on("OrderSubmitted", move |_: Value, evt, _ctx| { rows.lock() .unwrap() .get_mut(&evt.aggregate_id) .expect("placed") .status = "submitted".into(); async { Ok(()) } }); let rows = Arc::clone(&orders); orders_readmodel.query("GetOrder", move |data: Id, _qry, _ctx| { let row = rows.lock().unwrap().get(&data.id).cloned(); async move { Ok(row .ok_or_else(|| Rejected::new(format!("No order {}", data.id)).code("not_found"))?) } }); ``` and run both: read-models.ts: ```ts const kernable = new Kernable(); await Promise.all([kernable.listener(customersReadmodel).run(), kernable.listener(ordersReadmodel).run()]); ``` read_models.py: ```py threading.Thread(target=k.listener(customers_readmodel).run, daemon=True).start() k.listener(orders_readmodel).run() ``` read_models.rs: ```rs let kernable = Kernable::new(Options::default())?; let customers_readmodel = kernable.listener(&customers_readmodel); let orders_readmodel = kernable.listener(&orders_readmodel); tokio::join!(customers_readmodel.run(), orders_readmodel.run()); ``` A read model is rebuilt from the first event every time it starts; part 8 shows how to keep it in a file instead. Start it next to the worker (`node read-models.ts`, `.venv/bin/python read_models.py` or `cargo run -q --bin read_models`). ## Read your own writes part3.ts: ```ts const getOrder = (id: string) => kernable.query({ domain: "shop", type: "GetOrder", data: { id } }); await send("customer", "c1", "Register", { email: "ann@example.com", name: "Ann" }); await send("order", "o1", "Place", { customerId: "c1" }); await send("order", "o1", "AddItem", { sku: "apple", qty: 2 }); // The submit returned after the read model processed the event: the query sees it. console.log(await getOrder("o1")); console.log(await kernable.query({ domain: "shop", type: "GetCustomer", data: { id: "c1" } })); await getOrder("o9").catch((err) => console.log(`${err.code}: ${err.message}`)); ``` part3.py: ```py def get_order(id): return k.query("shop", "GetOrder", {"id": id}) send("customer", "c1", "Register", {"email": "ann@example.com", "name": "Ann"}) send("order", "o1", "Place", {"customerId": "c1"}) send("order", "o1", "AddItem", {"sku": "apple", "qty": 2}) # The submit returned after the read model processed the event: the query sees it. print(get_order("o1")) print(k.query("shop", "GetCustomer", {"id": "c1"})) try: get_order("o9") except Rejected as err: print(f"{err.code}: {err}") ``` part3.rs: ```rs let get_order = |id: &str| { kernable.query::(Query::new("shop", "GetOrder").data(json!({ "id": id }))) }; send( "customer", "c1", "Register", json!({ "email": "ann@example.com", "name": "Ann" }), ) .await?; send("order", "o1", "Place", json!({ "customerId": "c1" })).await?; send( "order", "o1", "AddItem", json!({ "sku": "apple", "qty": 2 }), ) .await?; // The submit returned after the read model processed the event: the query sees it. println!("{}", get_order("o1").await?); let customer: Value = kernable .query(Query::new("shop", "GetCustomer").data(json!({ "id": "c1" }))) .await?; println!("{customer}"); if let Err(Error::Rejected(r)) = get_order("o9").await { println!("{}: {}", r.code, r.message); } ``` TypeScript: ```text { id: 'o1', customerId: 'c1', status: 'draft', items: { apple: 2 } } { id: 'c1', email: 'ann@example.com', name: 'Ann' } not_found: No order o9 ``` Python: ```text {'id': 'o1', 'customerId': 'c1', 'status': 'draft', 'items': {'apple': 2}} {'id': 'c1', 'email': 'ann@example.com', 'name': 'Ann'} not_found: No order o9 ``` Rust: ```text {"id":"o1","customerId":"c1","status":"draft","items":{"apple":2}} {"id":"c1","email":"ann@example.com","name":"Ann"} not_found: No order o9 ``` The query right after `AddItem` already shows the apples. By default a submit **settles**: it returns only after every listener has processed the command's events. There is no polling and no "eventually". If you do not need that, submit with `wait: "committed"` and it returns once the events are stored. Next: [4. A reactor](./4-reactor). --- # 4. A reactor A **reactor** is a listener that acts on events: it sends emails, calls other systems, or sends follow-up commands. Here it confirms every submitted order to the customer's email address. ## Confirming an order The order gets a command and an event for the confirmation, in `domain.ts`: domain.ts: ```ts orderAggregate.command("Confirm", (s, data: { customerId: string; email: string }) => { if (s.status !== "submitted") throw new Rejected(`Cannot confirm a ${s.status} order`, "invalid_state"); return [{ type: "OrderConfirmed", data: { customerId: data.customerId, email: data.email } }]; }); ``` domain.py: ```py @order_aggregate.command("Confirm") def _(state, data): if state["status"] != "submitted": raise Rejected(f"Cannot confirm a {state['status']} order", "invalid_state") return [{"type": "OrderConfirmed", "data": {"customerId": data["customerId"], "email": data["email"]}}] ``` domain.rs: ```rs #[derive(Serialize, Deserialize)] #[serde(rename_all = "camelCase")] struct Confirm { customer_id: String, email: String, } fn confirm(order_aggregate: &Aggregate) { order_aggregate.command("Confirm", |s, data: Confirm, _cmd, _ctx| { if s.status != "submitted" { let message = format!("Cannot confirm a {} order", s.status); return Err(Rejected::new(message).code("invalid_state")); } Ok(vec![NewEvent::new("OrderConfirmed", data)]) }); } ``` and the orders read model learns about it, in `read-models.ts`: read-models.ts: ```ts ordersReadmodel.on("OrderConfirmed", (data: { email: string }, evt) => { const row = orders.get(evt.aggregateId)!; row.status = "confirmed"; row.confirmedTo = data.email; }); ``` read_models.py: ```py @orders_readmodel.on("OrderConfirmed") def _(data, evt): orders[evt.aggregate_id].update(status="confirmed", confirmedTo=data["email"]) ``` read_models.rs: ```rs let rows = Arc::clone(&orders); orders_readmodel.on("OrderConfirmed", move |data: Value, evt, _ctx| { let mut rows = rows.lock().unwrap(); let row = rows.get_mut(&evt.aggregate_id).expect("placed"); row.status = "confirmed".into(); row.confirmed_to = Some(data["email"].as_str().map(str::to_owned)); async { Ok(()) } }); ``` ## The reactor confirmations.ts: ```ts // Who this service is (part 6), and the domain's declarations, e.g. which fields are personal (part 7). const kernable = new Kernable({ context: { senderIdentity: "svc-confirmations" } }).use(shopDomain); const confirmationsReactor = shopDomain.reactor("confirmations") .on("OrderSubmitted", async (_data, evt, ctx) => { // The read models (order 0) have processed this event before the reactor (order 10) gets it. const order = await ctx.query<{ customerId: string }>("GetOrder", { id: evt.aggregateId }); const customer = await ctx.query<{ email: string }>("GetCustomer", { id: order.customerId }); // Same sender and correlation as the submit that caused the event; a rejection parks the event. await ctx.submit({ aggregateType: "order", aggregateId: evt.aggregateId, type: "Confirm", data: { customerId: order.customerId, email: customer.email }, id: `confirm-${evt.aggregateId}`, // a redelivered event confirms once }); }); await kernable.listener(confirmationsReactor).run(); ``` confirmations.py: ```py k = Kernable(context=RequestContext(sender_identity="svc-confirmations")).use(shop_domain) # parts 6 and 7 confirmations_reactor = shop_domain.reactor("confirmations") @confirmations_reactor.on("OrderSubmitted") def _(data, evt, ctx): # The read models (order 0) have processed this event before the reactor (order 10) gets it. order = ctx.query("GetOrder", {"id": evt.aggregate_id}) customer = ctx.query("GetCustomer", {"id": order["customerId"]}) # Same sender and correlation as the submit that caused the event; a rejection parks the event. ctx.submit("order", evt.aggregate_id, "Confirm", {"customerId": order["customerId"], "email": customer["email"]}, id=f"confirm-{evt.aggregate_id}") # a redelivered event confirms once k.listener(confirmations_reactor).run() ``` confirmations.rs: ```rs // Who this service is (part 6), and the domain's declarations, e.g. which fields are personal (part 7). let me = RequestContext { sender_identity: "svc-confirmations".into(), ..Default::default() }; let shop_domain = shop().domain; let kernable = Kernable::new(Options::default().context(me))?.with_domain(&shop_domain); #[derive(Deserialize)] #[serde(rename_all = "camelCase")] struct OrderRow { customer_id: String, } #[derive(Deserialize)] struct CustomerRow { email: String, } let confirmations_reactor = shop_domain.reactor("confirmations").on( "OrderSubmitted", |_: Value, evt, ctx| async move { // The read models (order 0) have processed this event before the reactor (order 10) gets it. let get = |query_type, id: &str| Query::new("shop", query_type).data(json!({ "id": id })); let order: OrderRow = ctx.query(get("GetOrder", &evt.aggregate_id)).await?; let customer: CustomerRow = ctx.query(get("GetCustomer", &order.customer_id)).await?; // Same sender and correlation as the submit that caused the event; a rejection parks the event. let confirm = Submit::new("shop", "order", &evt.aggregate_id, "Confirm") .data(json!({ "customerId": order.customer_id, "email": customer.email })) .id(format!("confirm-{}", evt.aggregate_id)); // a redelivered event confirms once ctx.submit(confirm).await?; Ok(()) }, ); kernable.listener(&confirmations_reactor).run().await; ``` Three things make it safe: - **Order.** Read models have order 0, reactors order 10. A listener gets an event only after every listener with a lower order has processed it, so the reactor's queries always find the order and the customer. - **A fixed command ID.** Delivery is at least once: after a crash the reactor may get the event again. With `confirm-` as the ID, the second confirmation returns the first one's outcome. `ctx.submit` refuses a follow-up without one. - **`ctx.submit`** gives the follow-up command the sender and correlation of the command that caused the event (`ctx.follow()`), so you can trace a confirmation back to the submit. Its queries (`ctx.query`) stay the reactor's own. If the confirmation is rejected, `ctx.submit` throws, and the reactor **parks** the event: it moves on to the next order, and the kernel keeps the event with the rejection until someone looks at it (`kernable listeners parked`, the [UI](../../how-to/monitor#the-ui)) and retries or dismisses it. A handler that fails for another reason (a bug, a mail server that is down) gets the event again a few times first; while the kernel is unreachable it simply waits. See [Failures](../../concepts/failures). Its own submit settles too, but only on the listeners before it: a listener handles one event at a time, so the kernel never makes it wait for itself. Start it (`node confirmations.ts` or `.venv/bin/python confirmations.py`) and submit an order: part4.ts: ```ts await send("customer", "c1", "Register", { email: "ann@example.com", name: "Ann" }); await send("order", "o1", "Place", { customerId: "c1" }); await send("order", "o1", "AddItem", { sku: "apple", qty: 2 }); await send("order", "o1", "Submit"); // The submit settled after the reactor, which confirmed the order before it returned. console.log(await kernable.query({ domain: "shop", type: "GetOrder", data: { id: "o1" } })); ``` part4.py: ```py send("customer", "c1", "Register", {"email": "ann@example.com", "name": "Ann"}) send("order", "o1", "Place", {"customerId": "c1"}) send("order", "o1", "AddItem", {"sku": "apple", "qty": 2}) send("order", "o1", "Submit") # The submit settled after the reactor, which confirmed the order before it returned. print(k.query("shop", "GetOrder", {"id": "o1"})) ``` part4.rs: ```rs send( "customer", "c1", "Register", json!({ "email": "ann@example.com", "name": "Ann" }), ) .await?; send("order", "o1", "Place", json!({ "customerId": "c1" })).await?; send( "order", "o1", "AddItem", json!({ "sku": "apple", "qty": 2 }), ) .await?; send("order", "o1", "Submit", Value::Null).await?; // The submit settled after the reactor, which confirmed the order before it returned. let order: Value = kernable .query(Query::new("shop", "GetOrder").data(json!({ "id": "o1" }))) .await?; println!("{order}"); ``` TypeScript: ```text { id: 'o1', customerId: 'c1', status: 'confirmed', items: { apple: 2 }, confirmedTo: 'ann@example.com' } ``` Python: ```text {'id': 'o1', 'customerId': 'c1', 'status': 'confirmed', 'items': {'apple': 2}, 'confirmedTo': 'ann@example.com'} ``` Rust: ```text {"id":"o1","customerId":"c1","status":"confirmed","items":{"apple":2},"confirmedTo":"ann@example.com"} ``` The `Submit` settled after the reactor, which had confirmed the order by then: the query shows it confirmed. Next: [5. A second language](./5-languages). --- # 5. A second language So far every process was in one language. Nothing ties them together but the kernel's protocol, so each can be in another: the worker in Python, the read models in TypeScript, the reactor in Rust. Stop your processes and start them mixed, from the files of parts 2 to 4: ```sh .venv/bin/python worker.py & node read-models.ts & cargo run -q --bin confirmations & node part4.ts ``` The output is the same as in part 4 for the language of `part4`. kernable's tests run the shop in three mixes, each language once in every role. What a language needs to take part: - **Its own definitions.** `domain.ts`, `domain.py` and `domain.rs` each declare the shop's aggregates and, in part 7, its personal fields. A read model in Python that reads a TypeScript worker's events needs no aggregate, only event handlers; but declarations that change how events are stored or read (personal fields, event versions) must exist in every language that writes or reads them. - **JSON payloads.** Commands, events and query answers are JSON, so a `dict` in Python, an object in TypeScript and a serde struct in Rust are the same data. - **An SDK.** TypeScript, Python and Rust have one. Any other language can join with the gRPC protocol and the signing rules in the [reference](../../reference/protocol); test vectors check an implementation. Next: [6. Authorization](./6-authorization). --- # 6. Authorization So far any customer could do anything. Now customers may act only on themselves and their own orders. Grants decide what a **key**, that is a process, may do: send `shop.Register`, run the shop's commands, read its events. They know nothing about the people using your shop. Who the **person** behind a request is, is business data: the request's **request context**, signed with the request, carries it as `senderIdentity`. A **middleware** checks every command and query against it before the kernel accepts it. ## The guard A middleware is a listener with a check: it builds its knowledge from events, like a read model, and answers every check from it. guard.ts: ```ts const SERVICES = new Set(["svc-confirmations"]); // What a business process may send (part 10): it acts as `process:`. const PROCESSES: Record = { "process:shop.fulfilment": ["Ship", "Cancel"] }; const registered = new Set(); const ownerOf = new Map(); // order -> customer const guardReadmodel = shopDomain.readModel("guard") .on("CustomerRegistered", (_data, evt) => void registered.add(evt.aggregateId)) .on("CustomerErased", (_data, evt) => void registered.delete(evt.aggregateId)) .on("OrderPlaced", (data: { customerId: string }, evt) => void ownerOf.set(evt.aggregateId, data.customerId)) .middleware({ types: ["shop.*"] }, (check) => { const who = check.request.senderIdentity ?? ""; if (check.type === "Register") return; // anyone may sign up if (check.kind === "query" && SERVICES.has(who)) return; // services may read if (check.kind === "command" && PROCESSES[who]?.includes(check.type)) return; // Files and cache requests carry no data: they have no owner, so only the rules above pass them. const data = (check.data ?? {}) as { id?: string; customerId?: string }; const target = check.kind === "command" ? check.aggregateId! : data.id!; const owner = check.type === "Place" ? data.customerId : check.type === "GetCustomer" || check.aggregateType === "customer" ? target : ownerOf.get(target); if (!registered.has(who) || owner !== who) throw new Rejected(`${check.type} is not allowed for ${who || "anonymous"}`, "forbidden"); }); await new Kernable().listener(guardReadmodel).run(); ``` guard.py: ```py SERVICES = {"svc-confirmations"} # What a business process may send (part 10): it acts as `process:`. PROCESSES = {"process:shop.fulfilment": {"Ship", "Cancel"}} registered: set[str] = set() owner_of: dict[str, str] = {} # order -> customer guard_readmodel = shop_domain.read_model("guard") @guard_readmodel.on("CustomerRegistered") def _(data, evt): registered.add(evt.aggregate_id) @guard_readmodel.on("CustomerErased") def _(data, evt): registered.discard(evt.aggregate_id) @guard_readmodel.on("OrderPlaced") def _(data, evt): owner_of[evt.aggregate_id] = data["customerId"] @guard_readmodel.middleware(["shop.*"]) def _(check: Check): who = check.request.sender_identity if check.type == "Register": return # anyone may sign up if check.kind == "query" and who in SERVICES: return # services may read if check.kind == "command" and check.type in PROCESSES.get(who, ()): return data = check.data or {} # files and cache requests carry none: they have no owner target = check.aggregate_id if check.kind == "command" else data.get("id") if check.type == "Place": owner = check.data.get("customerId") elif check.type == "GetCustomer" or check.aggregate_type == "customer": owner = target else: owner = owner_of.get(target) if who not in registered or owner != who: raise Rejected(f"{check.type} is not allowed for {who or 'anonymous'}", "forbidden") Kernable().listener(guard_readmodel).run() ``` guard.rs: ```rs const SERVICES: [&str; 1] = ["svc-confirmations"]; // What a business process may send (part 10): it acts as `process:`. const PROCESSES: [(&str, &[&str]); 1] = [("process:shop.fulfilment", &["Ship", "Cancel"])]; let registered = Arc::new(Mutex::new(HashSet::::new())); let owner_of = Arc::new(Mutex::new(HashMap::::new())); // order -> customer let shop_domain = shop().domain; let guard_readmodel = shop_domain.read_model("guard"); let r = Arc::clone(®istered); guard_readmodel.on("CustomerRegistered", move |_: Value, evt, _ctx| { r.lock().unwrap().insert(evt.aggregate_id); async { Ok(()) } }); let r = Arc::clone(®istered); guard_readmodel.on("CustomerErased", move |_: Value, evt, _ctx| { r.lock().unwrap().remove(&evt.aggregate_id); async { Ok(()) } }); let o = Arc::clone(&owner_of); guard_readmodel.on("OrderPlaced", move |data: Value, evt, _ctx| { let customer = data["customerId"].as_str().unwrap_or_default().to_owned(); o.lock().unwrap().insert(evt.aggregate_id, customer); async { Ok(()) } }); let (r, o) = (Arc::clone(®istered), Arc::clone(&owner_of)); guard_readmodel.middleware(MiddlewareOptions::new(&["shop.*"]), move |check| { let who = check.request.sender_identity.clone(); let allowed = check.type_name == "Register" // anyone may sign up || (check.kind == CheckKind::Query && SERVICES.contains(&who.as_str())) // services may read || (check.kind == CheckKind::Command && PROCESSES.iter().any(|(process, types)| *process == who && types.contains(&check.type_name.as_str()))) || { let field = |name: &str| check.data[name].as_str().map(str::to_owned); let target = match check.kind { CheckKind::Command | CheckKind::Cancel => check.aggregate_id.clone(), CheckKind::Query => field("id"), // The shop keeps no files: no customer's to protect. CheckKind::Data | CheckKind::Cache => None, }; let owner = if check.type_name == "Place" { field("customerId") } else if check.type_name == "GetCustomer" || check.aggregate_type.as_deref() == Some("customer") { target } else { target.and_then(|t| o.lock().unwrap().get(&t).cloned()) }; r.lock().unwrap().contains(&who) && owner.as_deref() == Some(who.as_str()) }; let who = if who.is_empty() { "anonymous".to_owned() } else { who }; let refusal = format!("{} is not allowed for {who}", check.type_name); async move { if !allowed { return Err(Rejected::new(refusal).code("forbidden").into()); } Ok(()) } }); let kernable = Kernable::new(Options::default())?; kernable.listener(&guard_readmodel).run().await; ``` The check sees the command or query, with its data and request context, before the kernel stores or routes it. Throwing `Rejected` refuses it; a refused command is never stored. Middleware runs in a chain by `order`, and the kernel asks a middleware only once it has processed every event committed before, so it never decides on stale knowledge. The guard's key needs the `check` action on the types it checks; `app`, which `--dev` lets do everything, has it. The registration is kept by the kernel: **while the guard is not running, every request it covers is refused**. Authorization fails closed. ## Try it Start the guard (`node guard.ts`, `.venv/bin/python guard.py` or `cargo run -q --bin guard`) and send requests as different customers: part6.ts: ```ts // Who sends a request is part of its signed request context. const as = (senderIdentity: string) => ({ context: { senderIdentity } }); const send = (who: string, aggregateType: string, aggregateId: string, type: string, data?: object) => kernable.submit({ domain: "shop", aggregateType, aggregateId, type, data, ...as(who) }); show("c1 registers", await send("c1", "customer", "c1", "Register", { email: "ann@example.com", name: "Ann" })); show("c2 registers", await send("c2", "customer", "c2", "Register", { email: "bob@example.com", name: "Bob" })); show("c1 places o1", await send("c1", "order", "o1", "Place", { customerId: "c1" })); show("c2 adds to o1", await send("c2", "order", "o1", "AddItem", { sku: "apple", qty: 1 })); show("c2 places for c1", await send("c2", "order", "o2", "Place", { customerId: "c1" })); const getOrder = (who: string) => kernable.query({ domain: "shop", type: "GetOrder", data: { id: "o1" }, ...as(who) }); console.log("c1 reads o1:", await getOrder("c1")); await getOrder("c2").catch((err) => console.log(`c2 reads o1: ${err.code} (${err.message})`)); ``` part6.py: ```py # Who sends a request is part of its signed request context. def send(who, aggregate_type, aggregate_id, type, data=None): return k.submit("shop", aggregate_type, aggregate_id, type, data, context=RequestContext(sender_identity=who)) show("c1 registers", send("c1", "customer", "c1", "Register", {"email": "ann@example.com", "name": "Ann"})) show("c2 registers", send("c2", "customer", "c2", "Register", {"email": "bob@example.com", "name": "Bob"})) show("c1 places o1", send("c1", "order", "o1", "Place", {"customerId": "c1"})) show("c2 adds to o1", send("c2", "order", "o1", "AddItem", {"sku": "apple", "qty": 1})) show("c2 places for c1", send("c2", "order", "o2", "Place", {"customerId": "c1"})) def get_order(who): return k.query("shop", "GetOrder", {"id": "o1"}, context=RequestContext(sender_identity=who)) print("c1 reads o1:", get_order("c1")) try: get_order("c2") except Rejected as err: print(f"c2 reads o1: {err.code} ({err})") ``` part6.rs: ```rs // Who sends a request is part of its signed request context. let as_ = |who: &str| RequestContext { sender_identity: who.into(), ..Default::default() }; let send = |who, aggregate_type, aggregate_id, command_type, data: Value| { let command = Submit::new("shop", aggregate_type, aggregate_id, command_type) .data(data) .context(as_(who)); kernable.submit(command) }; show( "c1 registers", send( "c1", "customer", "c1", "Register", json!({ "email": "ann@example.com", "name": "Ann" }), ) .await?, ); show( "c2 registers", send( "c2", "customer", "c2", "Register", json!({ "email": "bob@example.com", "name": "Bob" }), ) .await?, ); show( "c1 places o1", send("c1", "order", "o1", "Place", json!({ "customerId": "c1" })).await?, ); show( "c2 adds to o1", send( "c2", "order", "o1", "AddItem", json!({ "sku": "apple", "qty": 1 }), ) .await?, ); show( "c2 places for c1", send("c2", "order", "o2", "Place", json!({ "customerId": "c1" })).await?, ); let get_order = |who| { kernable.query::( Query::new("shop", "GetOrder") .data(json!({ "id": "o1" })) .context(as_(who)), ) }; println!("c1 reads o1: {}", get_order("c1").await?); if let Err(Error::Rejected(r)) = get_order("c2").await { println!("c2 reads o1: {} ({})", r.code, r.message); } ``` TypeScript: ```text c1 registers: committed c2 registers: committed c1 places o1: committed c2 adds to o1: forbidden (AddItem is not allowed for c2) c2 places for c1: forbidden (Place is not allowed for c2) c1 reads o1: { id: 'o1', customerId: 'c1', status: 'draft', items: {} } c2 reads o1: forbidden (GetOrder is not allowed for c2) ``` Python: ```text c1 registers: committed c2 registers: committed c1 places o1: committed c2 adds to o1: forbidden (AddItem is not allowed for c2) c2 places for c1: forbidden (Place is not allowed for c2) c1 reads o1: {'id': 'o1', 'customerId': 'c1', 'status': 'draft', 'items': {}} c2 reads o1: forbidden (GetOrder is not allowed for c2) ``` Rust: ```text c1 registers: committed c2 registers: committed c1 places o1: committed c2 adds to o1: forbidden (AddItem is not allowed for c2) c2 places for c1: forbidden (Place is not allowed for c2) c1 reads o1: {"id":"o1","customerId":"c1","status":"draft","items":{}} c2 reads o1: forbidden (GetOrder is not allowed for c2) ``` The reactor still confirms orders: its follow-up commands carry the customer's identity (`ctx.submit` follows the event), and its own queries (`ctx.query`) come from `svc-confirmations`, a service the guard lets read. The kernel checks that the request was signed by a key with a grant to send it, not who `senderIdentity` names: any process that may submit can claim any identity. So in a real application the identity comes from your authentication, for example a verified token at your API, and only processes that verify it get a grant to submit. Part 8 gives each service its own key, so the reactor's key may submit nothing but `shop.Confirm`. Next: [7. Personal data](./7-personal-data). --- # 7. Personal data Events are kept forever and form a signed hash chain: you cannot change or delete one. But customers have a right to have their personal data erased, and passwords should not be stored at all. kernable keeps such values in a **vault** next to the log: - The log stores a reference instead of the value: `{"$vault": "3fa9…"}`. The reference is salted, so it reveals nothing about the value, also once the value is gone. - The vault keeps the values encrypted, and deletes them for real. - **Personal** values belong to a person (a subject) and are deleted when an event erases that person. - **Secret** values, like passwords, are deleted as soon as their command completed. The kernel you started in part 1 already has a vault: `--dev` turned it on with `vault.key`, and it keeps the values in `vault.db`. ## Declare what is personal In `domain.ts`, declare per command and event which fields go to the vault, and for whom; add passwords and erasure to the customer: domain.ts: ```ts // Personal data goes to the kernel's vault; the signed log keeps references only. shopDomain.command("Register").personal(["email", "name"], (_d, id) => id); shopDomain.event("CustomerRegistered", 1).personal(["email", "name"], (_d, id) => id); shopDomain.command("Confirm").personal(["email"], (d: { customerId: string }) => d.customerId); shopDomain.event("OrderConfirmed", 1).personal(["email"], (d: { customerId: string }) => d.customerId); // Passwords are secrets: the handler sees them once; the vault deletes them when it is done. shopDomain.command("SetPassword").secret(["password"]); shopDomain.command("SignIn").secret(["password"]); shopDomain.event("PasswordSet", 1).personal(["hash", "salt"], (_d, id) => id); // Erasing a customer deletes their personal data everywhere, in every stream. shopDomain.event("CustomerErased", 1).erases((_d, id) => id); const hash = (password: string, salt: string) => scryptSync(password, salt, 32).toString("hex"); customerAggregate .apply("PasswordSet", (s, data: { hash: string; salt: string }) => { s.passwordHash = data.hash; s.salt = data.salt; }) .apply("CustomerErased", (s) => { delete s.passwordHash; delete s.salt; s.registered = false; }) .command("SetPassword", (s, data: { password: string }, _cmd, ctx) => { if (!s.registered) throw new Rejected("No such customer", "not_found"); if ((data.password ?? "").length < 8) throw new Rejected("At least 8 characters", "invalid"); const salt = ctx.id(); // deterministic: the command may run again return [{ type: "PasswordSet", data: { hash: hash(data.password, salt), salt } }]; }) .command("SignIn", (s, data: { password: string }) => { if (!s.passwordHash || hash(data.password ?? "", s.salt!) !== s.passwordHash) throw new Rejected("Wrong customer or password", "denied"); return [{ type: "SignedIn" }]; }) .command("Erase", (s) => { if (!s.registered) throw new Rejected("No such customer", "not_found"); return [{ type: "CustomerErased" }]; }); ``` domain.py: ```py # Personal data goes to the kernel's vault; the signed log keeps references only. shop_domain.command("Register").personal(["email", "name"], lambda d, customer_id: customer_id) shop_domain.event("CustomerRegistered", 1).personal(["email", "name"], lambda d, customer_id: customer_id) shop_domain.command("Confirm").personal(["email"], lambda d: d["customerId"]) shop_domain.event("OrderConfirmed", 1).personal(["email"], lambda d: d["customerId"]) # Passwords are secrets: the handler sees them once; the vault deletes them when it is done. shop_domain.command("SetPassword").secret(["password"]) shop_domain.command("SignIn").secret(["password"]) shop_domain.event("PasswordSet", 1).personal(["hash", "salt"], lambda d, customer_id: customer_id) # Erasing a customer deletes their personal data everywhere, in every stream. shop_domain.event("CustomerErased", 1).erases(lambda d, customer_id: customer_id) def password_hash(password: str, salt: str) -> str: return hashlib.scrypt(password.encode(), salt=salt.encode(), n=2**14, r=8, p=1, dklen=32).hex() @customer_aggregate.apply("PasswordSet") def _(state, data): state["password_hash"] = data["hash"] state["salt"] = data["salt"] @customer_aggregate.apply("CustomerErased") def _(state, data): state.pop("password_hash", None) state.pop("salt", None) state["registered"] = False @customer_aggregate.command("SetPassword") def _(state, data, cmd, ctx): if not state["registered"]: raise Rejected("No such customer", "not_found") if len(data.get("password") or "") < 8: raise Rejected("At least 8 characters", "invalid") salt = ctx.id() # deterministic: the command may run again return [{"type": "PasswordSet", "data": {"hash": password_hash(data["password"], salt), "salt": salt}}] @customer_aggregate.command("SignIn") def _(state, data): if "password_hash" not in state or password_hash(data.get("password") or "", state["salt"]) != state["password_hash"]: raise Rejected("Wrong customer or password", "denied") return [{"type": "SignedIn"}] @customer_aggregate.command("Erase") def _(state, data): if not state["registered"]: raise Rejected("No such customer", "not_found") return [{"type": "CustomerErased"}] ``` domain.rs: ```rs #[derive(Deserialize)] struct Password { password: Option, } #[derive(Serialize, Deserialize)] struct PasswordSet { hash: String, salt: String, } fn vault(shop_domain: &Domain, customer_aggregate: &Aggregate) { let aggregate = |_: &Value, id: &str| id.to_owned(); let customer_id = |d: &Value, _: &str| d["customerId"].as_str().unwrap_or_default().to_owned(); // Personal data goes to the kernel's vault; the signed log keeps references only. shop_domain .command("Register") .personal(&["email", "name"], aggregate); shop_domain .event("CustomerRegistered", 1) .personal(&["email", "name"], aggregate); shop_domain .command("Confirm") .personal(&["email"], customer_id); shop_domain .event("OrderConfirmed", 1) .personal(&["email"], customer_id); // Passwords are secrets: the handler sees them once; the vault deletes them when it is done. shop_domain.command("SetPassword").secret(&["password"]); shop_domain.command("SignIn").secret(&["password"]); shop_domain .event("PasswordSet", 1) .personal(&["hash", "salt"], aggregate); // Erasing a customer deletes their personal data everywhere, in every stream. shop_domain.event("CustomerErased", 1).erases(aggregate); customer_aggregate.apply("PasswordSet", |s, data: PasswordSet, _evt, _ctx| { s.password_hash = Some(data.hash); s.salt = Some(data.salt); }); customer_aggregate.apply("CustomerErased", |s, _: Value, _evt, _ctx| { *s = Customer::default() }); customer_aggregate.command("SetPassword", |s, data: Password, _cmd, ctx| { if !s.registered { return Err(Rejected::new("No such customer").code("not_found")); } let password = data.password.unwrap_or_default(); if password.chars().count() < 8 { return Err(Rejected::new("At least 8 characters").code("invalid")); } let salt = ctx.id(); // deterministic: the command may run again Ok(vec![NewEvent::new( "PasswordSet", PasswordSet { hash: hash(&password, &salt), salt, }, )]) }); customer_aggregate.command("SignIn", |s, data: Password, _cmd, _ctx| { let password = data.password.unwrap_or_default(); match (&s.password_hash, &s.salt) { (Some(stored), Some(salt)) if hash(&password, salt) == *stored => { Ok(vec![NewEvent::new("SignedIn", ())]) } _ => Err(Rejected::new("Wrong customer or password").code("denied")), } }); customer_aggregate.command("Erase", |s, _: Value, _cmd, _ctx| { if !s.registered { return Err(Rejected::new("No such customer").code("not_found")); } Ok(vec![NewEvent::new("CustomerErased", ())]) }); } /// scrypt with the parameters of the TypeScript and Python shops, so their hashes match. fn hash(password: &str, salt: &str) -> String { let params = scrypt::Params::new(14, 8, 1, 32).expect("valid scrypt parameters"); let mut out = [0u8; 32]; scrypt::scrypt(password.as_bytes(), salt.as_bytes(), ¶ms, &mut out).expect("32 bytes fit"); out.iter().map(|b| format!("{b:02x}")).collect() } ``` The handlers do not change: they still see `email` and `password` in clear. The SDK moves the fields to the vault before a command or event is signed, and puts them back for the processes allowed to read them. Two actions of a grant decide who that is: - `vault_read` lists the events and commands whose vault values a listener or middleware gets in clear. A read model that shows email addresses needs it; a guard that only compares IDs does not, and sees the references. - `vault_erase` lists the erasing events a worker may write. Erasing is final, so grant it only for the events meant for it: `vault_erase=shop.CustomerErased`. `app`, which `--dev` lets do everything, has both. Part 8 grants them per service. Read models hold values too, so they must forget an erased customer. In `read-models.ts`: read-models.ts: ```ts // An erased customer's personal data must not outlive the erasure in a read model either. customersReadmodel.on("CustomerErased", (_data, evt) => void customers.delete(evt.aggregateId)); ordersReadmodel.on("CustomerErased", (_data, evt) => { for (const row of orders.values()) if (row.customerId === evt.aggregateId && row.confirmedTo) row.confirmedTo = null; }); ``` read_models.py: ```py # An erased customer's personal data must not outlive the erasure in a read model either. @customers_readmodel.on("CustomerErased") def _(data, evt): customers.pop(evt.aggregate_id, None) @orders_readmodel.on("CustomerErased") def _(data, evt): for row in orders.values(): if row["customerId"] == evt.aggregate_id and row.get("confirmedTo"): row["confirmedTo"] = None ``` read_models.rs: ```rs // An erased customer's personal data must not outlive the erasure in a read model either. let rows = Arc::clone(&customers); customers_readmodel.on("CustomerErased", move |_: Value, evt, _ctx| { rows.lock().unwrap().remove(&evt.aggregate_id); async { Ok(()) } }); let rows = Arc::clone(&orders); orders_readmodel.on("CustomerErased", move |_: Value, evt, _ctx| { for row in rows.lock().unwrap().values_mut() { if row.customer_id == evt.aggregate_id && row.confirmed_to.is_some() { row.confirmed_to = Some(None); } } async { Ok(()) } }); ``` Clients that send commands with personal or secret fields need the declarations: create them with `.use(shopDomain)`, as the reactor in part 4 already does. ## Try it Restart every process, then: part7.ts: ```ts // use(shopDomain): the client moves the fields the domain declares personal or secret to the vault. const kernable = new Kernable({ context: { senderIdentity: "c1" } }).use(shopDomain); const customer = (type: string, data?: object) => kernable.submit({ domain: "shop", aggregateType: "customer", aggregateId: "c1", type, data }); show("register", await customer("Register", { email: "ann@example.com", name: "Ann" })); show("set password", await customer("SetPassword", { password: "correct horse" })); show("sign in, wrong password", await customer("SignIn", { password: "wrong horse" })); show("sign in", await customer("SignIn", { password: "correct horse" })); console.log(await kernable.query({ domain: "shop", type: "GetCustomer", data: { id: "c1" } })); show("erase", await customer("Erase")); // Erased: the customer's data is gone, and with it the right to act. await kernable.query({ domain: "shop", type: "GetCustomer", data: { id: "c1" } }).catch((err) => console.log(`read: ${err.code}`)); ``` part7.py: ```py # use(shop_domain): the client moves the fields the domain declares personal or secret to the vault. k = Kernable(context=RequestContext(sender_identity="c1")).use(shop_domain) def customer(type, data=None): return k.submit("shop", "customer", "c1", type, data) show("register", customer("Register", {"email": "ann@example.com", "name": "Ann"})) show("set password", customer("SetPassword", {"password": "correct horse"})) show("sign in, wrong password", customer("SignIn", {"password": "wrong horse"})) show("sign in", customer("SignIn", {"password": "correct horse"})) print(k.query("shop", "GetCustomer", {"id": "c1"})) show("erase", customer("Erase")) # Erased: the customer's data is gone, and with it the right to act. try: k.query("shop", "GetCustomer", {"id": "c1"}) except Rejected as err: print(f"read: {err.code}") ``` part7.rs: ```rs // with_domain: the client moves the fields the domain declares personal or secret to the vault. let me = RequestContext { sender_identity: "c1".into(), ..Default::default() }; let kernable = Kernable::new(Options::default().context(me))?.with_domain(&shop().domain); let customer = |command_type, data: Value| { kernable.submit(Submit::new("shop", "customer", "c1", command_type).data(data)) }; let read = || kernable.query::(Query::new("shop", "GetCustomer").data(json!({ "id": "c1" }))); show( "register", customer( "Register", json!({ "email": "ann@example.com", "name": "Ann" }), ) .await?, ); show( "set password", customer("SetPassword", json!({ "password": "correct horse" })).await?, ); show( "sign in, wrong password", customer("SignIn", json!({ "password": "wrong horse" })).await?, ); show( "sign in", customer("SignIn", json!({ "password": "correct horse" })).await?, ); println!("{}", read().await?); show("erase", customer("Erase", Value::Null).await?); // Erased: the customer's data is gone, and with it the right to act. if let Err(Error::Rejected(r)) = read().await { println!("read: {}", r.code); } ``` TypeScript: ```text register: committed set password: committed sign in, wrong password: denied (Wrong customer or password) sign in: committed { id: 'c1', email: 'ann@example.com', name: 'Ann' } erase: committed read: forbidden ``` Python: ```text register: committed set password: committed sign in, wrong password: denied (Wrong customer or password) sign in: committed {'id': 'c1', 'email': 'ann@example.com', 'name': 'Ann'} erase: committed read: forbidden ``` Rust: ```text register: committed set password: committed sign in, wrong password: denied (Wrong customer or password) sign in: committed {"id":"c1","email":"ann@example.com","name":"Ann"} erase: committed read: forbidden ``` After `Erase`, the customer's email and name are gone from the vault, in the customer's stream and in every order's confirmation alike. A read model rebuilt from the log reads `null` where they were. The events themselves stay, and the hash chain still verifies. Next: [8. Operate](./8-operate). --- # 8. Operate The shop works. This part runs it the way you would in production. ## The production kernel Every party signs what it sends with its own Ed25519 key. A signature says **who** sent a message, but anyone can make a key, so the kernel trusts one key only: your **root key**. The root key signs **grants**, and a grant says **what** a key may do: which actions (submit, run, listen, query, …) on which types. The kernel holds no list of keys or permissions; your processes present their grants. In part 1, `--dev` made one key, `app`, with a grant for everything, and left the root key next to it. In production: - Each service gets its **own key and a grant for only what it does**. A stolen key then can do only what its service could. - The **root key stays offline**. The kernel needs only its public half, `root.pub`, to check grants. You need the root key only to issue a grant or to revoke a key. - Over a network, the kernel speaks TCP with **mutual TLS**: every process presents a certificate from your CA. TLS decides who may connect; keys and grants still decide what a connection may do. For trying it out, `kernable certs new` makes a development CA and certificates; in production use your own PKI. ```sh mkdir -p ~/shop-prod && cd ~/shop-prod ``` Make the root key where it will stay: ```sh # The root key lives on a machine that stays offline; here the directory offline/ stands in for it. mkdir -p offline && kernable keys new offline/root >/dev/null ``` Then, on the kernel's machine, the kernel's keys, a key per service, and the certificates. `api` stands for your scripts and your API, the processes that submit and query on behalf of customers: ```sh mkdir -p data keys && cp offline/root.pub keys/ kernable keys new keys/kernel >/dev/null kernable keys new keys/vault >/dev/null for service in worker read-models confirmations guard api; do kernable keys new keys/$service >/dev/null; done kernable certs new certs worker read-models confirmations guard api # CA, kernel, one per service ``` Bring the services' `.pub` files to the root key, grant each what it does, and bring the `.grant` files back next to the keys: ```sh kernable grant offline/root.key keys/worker.pub 'run=shop.*' 'vault_erase=shop.CustomerErased' kernable grant offline/root.key keys/read-models.pub 'listen=shop.*' 'answer=shop.*' 'vault_read=shop.*' kernable grant offline/root.key keys/confirmations.pub 'listen=shop.OrderSubmitted' \ 'query=shop.GetOrder,shop.GetCustomer' 'submit=shop.Confirm' kernable grant offline/root.key keys/guard.pub \ 'listen=shop.CustomerRegistered,shop.CustomerErased,shop.OrderPlaced' 'check=shop.*' kernable grant offline/root.key keys/api.pub 'submit=shop.*' 'query=shop.*' ``` | Key | May | Because | |---|---|---| | `worker` | run the shop's commands, erase with `CustomerErased` | it runs `Erase` | | `read-models` | read events, answer queries, read vault values | the customers model shows emails | | `confirmations` | read `OrderSubmitted`, ask two queries, send `Confirm` | that is all the reactor does | | `guard` | read three events, check every request | it compares IDs, so it needs no vault values | | `api` | submit commands, ask queries | the guard decides for which customer | A key and its grant are public; only the `.key` file is secret. Start the kernel: ```sh kernable start --root keys/root.pub --kernel-key keys/kernel.key \ --store sqlite:data/shop.db --snapshots sqlite:data/snapshots.db \ --vault-key keys/vault.key --vault-store sqlite:data/vault.db \ --listen 0.0.0.0:7233 \ --tls-cert certs/kernel.tls.crt --tls-key certs/kernel.tls.key --tls-client-ca certs/ca.tls.crt \ --metrics-listen 127.0.0.1:9233 ``` Besides TLS and the keys, four things differ from part 1: - **Snapshots** go to their own SQLite file. A worker sends a snapshot of an aggregate every 100 events, so a long-lived aggregate does not have to replay its whole history. Snapshots are a cache: delete them, and workers rebuild them. - **Metrics** and health checks are served on `--metrics-listen`; keep that address private. - **No UI.** `--dev` turned on the kernel's UI at http://127.0.0.1:7234; in production it is off unless you set `--ui-listen`. It has no login and listens on loopback only, so reach it through an SSH tunnel ([Monitor](../../how-to/monitor#the-ui)). - Paths point into `data/` and `keys/`. Back up `data/` (the log, the vault) and `keys/`, and keep a copy of the root key, offline too; `kernable verify` checks a log and its signatures at any time. Every flag can also be set as an environment variable (`KERNABLE_STORE`, `KERNABLE_TLS_CERT`, …), which suits a service manager or a container better than a long command line. ## Connect the processes The SDKs read the connection from the environment (part 1), so your processes run unchanged. Each one gets its own key and certificate; the SDK finds the grant next to the key: ```sh export KERNABLE_URL=kernel.example.internal:7233 KERNABLE_TLS_CA=certs/ca.tls.crt export KERNABLE_KEY=keys/worker.key KERNABLE_TLS_CERT=certs/worker.tls.crt KERNABLE_TLS_KEY=certs/worker.tls.key node worker.ts ``` Start `read-models`, `confirmations` and `guard` the same way with their keys, and your scripts with `api`. To keep the grant elsewhere, set `KERNABLE_GRANT` to its file. ## Change what a key may do The kernel stores no permissions; it decides by the newest grant it has seen for a key. So adding a service needs no change to the kernel: make its key, issue its grant, start it. To give a service more or fewer rights, issue a new grant for the same key and deploy it. Once the service presented it, the kernel holds the older one invalid and checks the service's open sessions against the new one. A grant cannot be withdrawn: after a restart, the kernel accepts an older grant again until it sees the newer one. When it must be gone for sure, rotate the key and revoke the old one. ## Rotate and revoke keys To **rotate a service's key**, make a new key, grant it, deploy the service with it, then revoke the old key. A listener's registration belongs to its key; to keep its position, the new grant names the key it succeeds: ```sh kernable keys new keys/read-models-2 >/dev/null kernable grant offline/root.key keys/read-models-2.pub --succeeds keys/read-models.pub \ 'listen=shop.*' 'answer=shop.*' 'vault_read=shop.*' ``` When the read models connect with `read-models-2`, the kernel moves their registration to it. To **revoke** a key, because it was replaced or leaked, sign a revocation with the root key and send it to the running kernel. It is the one command that needs the root key and the kernel at once; the revocation is valid for a minute, so it cannot be prepared offline: ```sh export KERNABLE_TLS_CA=certs/ca.tls.crt KERNABLE_TLS_CERT=certs/api.tls.crt KERNABLE_TLS_KEY=certs/api.tls.key kernable revoke kernel.example.internal:7233 offline/root.key keys/read-models.pub 2026-10-27T09:00:00Z ``` Over TCP, `kernable` connects with mutual TLS like every other process, with the certificate files in these variables. From then on the kernel refuses the key and ends its sessions. Give the time from which you no longer trust the key, for a leak the time it may have leaked: `kernable verify` flags everything it signed since. The kernel stores the revocation; it is never undone. The **kernel's key** signs checkpoints and revocations. Replace `keys/kernel.key` and send the kernel `SIGHUP`; it reads the key again without a restart. Keep the old `kernel.pub`: `kernable verify` needs every key the kernel ever had. The **root key** is rotated by trusting two for a while: start the kernel with `--root keys/root.pub --root keys/root-2.pub`, issue every grant again with the new root, then restart with the new root only. ## Read models that survive a restart A read model in memory reads every event again when it starts. With a **store** it keeps its rows in a file, writes them together with its position per batch, and continues where it left off: customers-persistent.ts: ```ts type CustomerRow = { id: string; email: string; name: string }; const customersReadmodel = shopDomain.readModel("customers", { store: "sqlite:customers.db", schema: "1" }) .on("CustomerRegistered", (data: { email: string; name: string }, evt, ctx) => { ctx.store!.set("customers", evt.aggregateId, { id: evt.aggregateId, email: data.email, name: data.name }); }) .on("CustomerErased", (_data, evt, ctx) => ctx.store!.remove("customers", evt.aggregateId)) .query("GetCustomer", (data: { id: string }, _qry, ctx) => { const row = ctx.store!.get("customers", data.id); if (!row) throw new Rejected(`No customer ${data.id}`, "not_found"); return row; }); await new Kernable().listener(customersReadmodel).run(); ``` customers_persistent.py: ```py customers_readmodel = shop_domain.read_model("customers", store="sqlite:customers.db", schema="1") @customers_readmodel.on("CustomerRegistered") def _(data, evt, ctx): ctx.store.set("customers", evt.aggregate_id, {"id": evt.aggregate_id, "email": data["email"], "name": data["name"]}) @customers_readmodel.on("CustomerErased") def _(data, evt, ctx): ctx.store.remove("customers", evt.aggregate_id) @customers_readmodel.query("GetCustomer") def _(data, qry, ctx): row = ctx.store.get("customers", data["id"]) if row is None: raise Rejected(f"No customer {data['id']}", "not_found") return row Kernable().listener(customers_readmodel).run() ``` customers_persistent.rs: ```rs #[derive(Serialize, Deserialize)] struct CustomerRow { id: String, email: String, name: String, } #[derive(Deserialize)] struct Id { id: String, } let shop_domain = shop().domain; let customers_readmodel = shop_domain .read_model("customers") .store("sqlite:customers.db", "1"); customers_readmodel.on("CustomerRegistered", |data: Value, evt, ctx| async move { let (email, name) = ( data["email"].as_str().unwrap_or_default(), data["name"].as_str().unwrap_or_default(), ); let row = CustomerRow { id: evt.aggregate_id.clone(), email: email.into(), name: name.into(), }; ctx.store .expect("a store") .set("customers", &evt.aggregate_id, &row)?; Ok(()) }); customers_readmodel.on("CustomerErased", |_: Value, evt, ctx| async move { ctx.store .expect("a store") .remove("customers", &evt.aggregate_id)?; Ok(()) }); customers_readmodel.query("GetCustomer", |data: Id, _qry, ctx| async move { let row: Option = ctx.store.expect("a store").get("customers", &data.id)?; Ok(row.ok_or_else(|| Rejected::new(format!("No customer {}", data.id)).code("not_found"))?) }); Kernable::new(Options::default())? .listener(&customers_readmodel) .run() .await; ``` When you change how the handlers store data, raise `schema`: the store is cleared and rebuilt from the first event. ## Readiness, deploys and shutdown - **Tell the kernel what each service consists of**, once per deploy: `kernable apply` with a key granted `manage`, or an SDK app (`kernable.app(...)`) that applies its membership when it starts. Then the kernel waits for exactly that, and what a new version no longer has stops counting ([Remove a listener, middleware or aggregate](../../how-to/remove-listener)). - `GET /ready` on the metrics address answers 200 once every registered aggregate has a worker connected and every registered listener is connected and has caught up; let your deploy wait for it. `GET /health` answers while the process runs, and `GET /metrics` serves Prometheus metrics. - **Deploy readers before writers.** When an event's format changes (a new event version), a process that does not know the new version stops at it instead of misreading it. - On SIGTERM the kernel **drains**: it refuses new requests, lets running ones finish for up to `--drain-timeout-ms`, then stores its state and exits. Commands that did not finish run right after the restart. - Restarting a worker or a listener loses nothing: commands go to another worker, and listeners continue after their last acknowledged event. - **Watch for parked events.** A reactor that gives up on an event (a rejected follow-up, or a handler that keeps failing) parks it and moves on; `kernable listeners parked` lists them, `kernable_listener_parked` counts them, and `kernable listeners retry` delivers them again once the cause is fixed ([Parked events](../../how-to/monitor#parked-events)). The [reference](../../reference/) lists every flag, metric and error. Next: [9. Accounts and sign-in](./9-sign-in). --- # 9. Accounts and sign-in In part 6 the guard trusted whatever `senderIdentity` a request named. Real customers sign in. In this part the shop gets an HTTP API with accounts, passwords and sessions, and the hand-written guard is replaced by [the kit](../../kits/): tenants, accounts, groups, invitation links, and a middleware that decides every request. The story: Ann opens her shop on your platform. She creates a group of customers and puts an invitation link on her website. Bob and Cleo sign up and join through it. Each may work only on their own orders. ## Copy the kit The kit is source code you copy into your project, not a dependency: ```sh [TypeScript] npx @kernable/sdk add kit npm install @node-rs/argon2 @simplewebauthn/server openid-client ``` ```sh [Python] .venv/bin/kernable-sdk add kit .venv/bin/pip install -r kit/requirements.txt ``` ```sh [Rust] cargo install kernable-sdk kernable-sdk add kit # then add the crate `kit` to your workspace and as a dependency ``` Stop the guard from part 6 and the confirmations reactor from part 4: from now on the kit's middleware decides, and a reactor needs an identity of its own under it (see [Identities](../../kits/identity)). ## Say what a query is about `GetOrder { id: "o1" }` is about order `o1`, but only the shop's code knows that `id` names an order. Declare it once, in the domain: domain.ts: ```ts // GetOrder { id } is about the order `id`: the SDK sends that order with the query, signed, and a // middleware decides on it as on a command's aggregate (part 9). shopDomain.query("GetOrder").target("order", "id"); ``` domain.py: ```py # GetOrder {"id": ...} is about the order `id`: the SDK sends that order with the query, signed, and a # middleware decides on it as on a command's aggregate (part 9). shop_domain.query("GetOrder").target("order", "id") ``` domain.rs: ```rs // GetOrder { id } is about the order `id`: the SDK sends that order with the query, signed, and a // middleware decides on it as on a command's aggregate (part 9). fn target(shop_domain: &Domain) { shop_domain.query("GetOrder").target("order", "id"); } ``` Now a rule can say "only the order's owner may read it", for queries as for commands. ## The API One new process, `api`, runs the kit's domains and its example HTTP server. It declares who owns an order, from the shop's own events: api.ts: ```ts // Who owns an order: the customer it was placed for. Declared before the kit's domains start. relation("shop.order", "owner").set("shop.OrderPlaced", (e) => e.data.customerId); ``` api.py: ```py # Who owns an order: the customer it was placed for. Declared before the kit's domains start. relation("shop.order", "owner").set("shop.OrderPlaced", lambda e: e.data["customer_id"]) ``` api.rs: ```rs // Who owns an order: the customer it was placed for. Declared before the kit's domains start. relation("shop.order", "owner").set("shop.OrderPlaced", |e| ids(&e.data["customerId"])); ``` The kit sends mail for email verification, password resets and invitations. In development, the API writes each mail to a file in `mail/` instead: api.ts: ```ts // In development the kit's mails go to files in mail/, named by address, instead of a mail server. mkdirSync("mail", { recursive: true }); useMailer({ async send(mail) { writeFileSync(`mail/${mail.to}`, mail.token); } }); ``` api.py: ```py # In development the kit's mails go to files in mail/, named by address, instead of a mail server. class FileMailer: def send(self, mail) -> None: with open(os.path.join("mail", mail.to), "w") as f: f.write(mail.token) os.makedirs("mail", exist_ok=True) use_mailer(FileMailer()) ``` api.rs: ```rs /// In development the kit's mails go to files in mail/, named by address, instead of a mail server. struct FileMailer; impl Mailer for FileMailer { fn send(&self, mail: Mail) { std::fs::write(format!("mail/{}", mail.to), mail.token).expect("mail/ is writable"); } } ``` Groups say which *types* someone may send. One rule of the shop is about the data instead: a new order's customer is whoever places it, or a customer could order in someone else's name. The API runs a small middleware of the shop for it. It comes after the kit's (order 0 against the kit's -30), so the sender is already checked when it compares: api.ts: ```ts // A rule of the shop that no group expresses: a new order's customer is whoever places it. A // middleware of the shop's own, after the kit's (order 0), so the sender is checked already. const ownOrders = shopDomain.readModel("own-orders").middleware({ types: ["shop.Place"] }, (check) => { if ((check.data as { customerId?: string }).customerId !== check.request.senderIdentity) { throw new Rejected("An order is placed by its own customer", "forbidden"); } }); void new Kernable().listener(ownOrders).run(); ``` api.py: ```py # A rule of the shop that no group expresses: a new order's customer is whoever places it. A # middleware of the shop's own, after the kit's (order 0), so the sender is checked already. own_orders = shop_domain.read_model("own-orders") @own_orders.middleware(["shop.Place"]) def _(check: Check): if check.data.get("customerId") != check.request.sender_identity: raise Rejected("An order is placed by its own customer", "forbidden") threading.Thread(target=Kernable().listener(own_orders).run, daemon=True).start() ``` api.rs: ```rs // A rule of the shop that no group expresses: a new order's customer is whoever places it. A // middleware of the shop's own, after the kit's (order 0), so the sender is checked already. let own_orders = shop().domain.read_model("own-orders"); own_orders.middleware(MiddlewareOptions::new(&["shop.Place"]), |check| { let own = check.data["customerId"].as_str() == Some(check.request.sender_identity.as_str()); async move { if !own { return Err(Rejected::new("An order is placed by its own customer") .code("forbidden") .into()); } Ok(()) } }); let rules = Kernable::new(Options::default())?; tokio::spawn(async move { rules.listener(&own_orders).run().await }); ``` Like the kit's middleware, it fails closed: while it is away, every `Place` is refused. Then it starts the kit, sets up the platform once (`bootstrap`: the platform tenant, open sign-up, the API's own identity and key) and serves HTTP on port 3000: api.ts: ```ts // The kit's domains: its aggregates, read models, reactors and the authorization middleware. const kit = new Kernable().app({ service: "kit", domains: kitDomains() }); void kit.run(); await kit.ready; // The API's client knows the shop's schemas too: GetOrder declares the order it is about. const kernable = new Kernable().use(...kitSchemas(), shopDomain); // Once: the platform tenant, the sign-up policy, and the API's own identity and key. const { api: service } = await bootstrap(kernable, { signUp: { accounts: "open", tenants: "active" } }); const api = new KitApi({ kernable, keys: newSigningKeys(), service }); exampleServer(api, kernable).listen(Number(process.env.PORT ?? 3000)); ``` api.py: ```py # The kit's domains: its aggregates, read models, reactors and the authorization middleware. kit = Kernable().app(service="kit", domains=kit_domains()) threading.Thread(target=kit.run, daemon=True).start() if not kit.ready.wait(30): sys.exit("the kit's service did not connect") # The API's client knows the shop's schemas too: GetOrder declares the order it is about. kernable = Kernable().use(*kit_domains(), shop_domain) # Once: the platform tenant, the sign-up policy, and the API's own identity and key. service = bootstrap(kernable, sign_up={"accounts": "open", "tenants": "active"})["api"] api = KitApi(kernable, new_signing_keys(), service) example_server(api, kernable, int(os.environ.get("PORT", "3000"))).serve_forever() ``` api.rs: ```rs // The kit's domains: its aggregates, read models, reactors and the authorization middleware. let domains = kit::kit_domains(); let mut definition = App::new().service("kit"); for d in &domains { definition = definition.domain(d); } let kit = Kernable::new(Options::default())?.app(definition)?; let running = kit.clone(); tokio::spawn(async move { running.run().await }); kit.ready().await; // The API's client knows the shop's schemas too: GetOrder declares the order it is about. let kernable = Kernable::new(Options::default())?.with_domain(&shop().domain); for d in &domains { kernable.with_domain(d); } // Once: the platform tenant, the sign-up policy, and the API's own identity and key. let policy = SignUpPolicy { accounts: AccountSignUp::Open, tenants: TenantSignUp::Active, }; let booted = bootstrap(&kernable, policy, "Platform", None).await?; let api = KitApi::new( kernable.clone(), new_signing_keys(), &booted.api_identity, &booted.api_key_id, ); let port = std::env::var("PORT").unwrap_or_else(|_| "3000".into()); let listener = tokio::net::TcpListener::bind(format!("127.0.0.1:{port}")).await?; axum::serve(listener, server::router(api, kernable)).await?; ``` Start the worker, the read models and the API (`node api.ts`, `.venv/bin/python api.py` or `cargo run -q --bin api`). The example server's routes: | Route | Does | |---|---| | `POST /register`, `POST /verify`, `POST /sign-in` | sign up, confirm the email with the mailed token, sign in for an access token | | `POST /tenants` | open a tenant the caller owns | | `POST /invitation-links`, `POST /invitation-links/join` | create an invitation link; join a tenant through one | | `GET /me` | the caller's request context in a tenant | | `POST /commands///` | send a command as the caller | | `POST /queries//` | ask a query as the caller | A request carries `Authorization: Bearer ` and, inside a tenant, `X-Tenant: `. The API turns them into the request context; it decides nothing itself. ## Try it A small client of the API, for the people of the story: part9.ts: ```ts // A small client of the API: a person's access token and, inside the shop, its tenant. type Person = { email: string; token?: string; tenant?: string }; async function call(method: string, path: string, body: object | null, as?: Person): Promise<[number, any]> { const headers: Record = { "content-type": "application/json" }; if (as?.token) headers.authorization = `Bearer ${as.token}`; if (as?.tenant) headers["x-tenant"] = as.tenant; const res = await fetch(`${API}${path}`, { method, headers, body: body ? JSON.stringify(body) : undefined }); return [res.status, res.status === 204 ? null : await res.json()]; } // Registers, confirms the email with the token from the mail (a file in development), signs in. async function signUp(name: string): Promise { const email = `${name.toLowerCase()}@example.com`; await call("POST", "/register", { email, name, password: "correct horse battery" }); await call("POST", "/verify", { token: readFileSync(`mail/${email}`, "utf8") }); const [status, signedIn] = await call("POST", "/sign-in", { email, password: "correct horse battery" }); console.log(`${name} signs up and signs in: ${status}`); return { email, token: signedIn.accessToken }; } ``` part9.py: ```py # A small client of the API: a person's access token and, inside the shop, its tenant. def call(method: str, path: str, body: dict | None, person: dict | None = None): headers = {"content-type": "application/json"} if person and person.get("token"): headers["authorization"] = f"Bearer {person['token']}" if person and person.get("tenant"): headers["x-tenant"] = person["tenant"] data = json.dumps(body).encode() if body is not None else None request = urllib.request.Request(f"{API}{path}", data=data, method=method, headers=headers) try: with urllib.request.urlopen(request) as res: status, raw = res.status, res.read() except urllib.error.HTTPError as err: status, raw = err.code, err.read() return status, (json.loads(raw) if raw and status != 204 else None) # Registers, confirms the email with the token from the mail (a file in development), signs in. def sign_up(name: str) -> dict: email = f"{name.lower()}@example.com" call("POST", "/register", {"email": email, "name": name, "password": "correct horse battery"}) with open(f"mail/{email}") as f: call("POST", "/verify", {"token": f.read()}) status, signed_in = call("POST", "/sign-in", {"email": email, "password": "correct horse battery"}) print(f"{name} signs up and signs in: {status}") return {"email": email, "token": signed_in["access_token"]} ``` part9.rs: ```rs /// A small client of the API: a person's access token and, inside the shop, its tenant. #[derive(Clone, Default)] struct Person { token: Option, tenant: Option, } struct Client { http: reqwest::Client, api: String, } impl Client { async fn call( &self, method: &str, path: &str, body: Option, person: &Person, ) -> Result<(u16, Value), BoxError> { let url = format!("{}{path}", self.api); let mut request = match method { "GET" => self.http.get(url), _ => self.http.post(url), }; if let Some(token) = &person.token { request = request.bearer_auth(token); } if let Some(tenant) = &person.tenant { request = request.header("x-tenant", tenant); } if let Some(body) = body { request = request.json(&body); } let res = request.send().await?; let status = res.status().as_u16(); let body = if status == 204 { Value::Null } else { res.json().await? }; Ok((status, body)) } /// Registers, confirms the email with the token from the mail (a file in development), signs in. async fn sign_up(&self, name: &str) -> Result { let email = format!("{}@example.com", name.to_lowercase()); let password = "correct horse battery"; let nobody = Person::default(); let register = json!({ "email": email, "name": name, "password": password }); self.call("POST", "/register", Some(register), &nobody) .await?; let token = std::fs::read_to_string(format!("mail/{email}"))?; self.call("POST", "/verify", Some(json!({ "token": token })), &nobody) .await?; let sign_in = json!({ "email": email, "password": password }); let (status, signed_in) = self .call("POST", "/sign-in", Some(sign_in), &nobody) .await?; println!("{name} signs up and signs in: {status}"); Ok(Person { token: signed_in["accessToken"].as_str().map(str::to_owned), tenant: None, }) } } ``` Ann opens her shop. A tenant is one customer of your platform, here Ann's shop; Ann is its owner and may do everything in it. She creates the group Customers: they may place orders, and add items to, submit and read only the orders they own. Then she creates an invitation link with that group: part9.ts: ```ts const ann = await signUp("Ann"); const [opened, shop] = await call("POST", "/tenants", { slug: "anns-shop", name: "Ann's shop" }, ann); ann.tenant = shop.tenant; console.log(`Ann opens her shop: ${opened}`); // Customers may place orders, and work only on orders they own. const [grouped] = await call("POST", "/commands/group/group/CreateGroup", { aggregateId: "customers", data: { name: "Customers", permissions: [ { pattern: "shop.Place" }, { pattern: "shop.AddItem", when: "owner" }, { pattern: "shop.Submit", when: "owner" }, { pattern: "shop.GetOrder", when: "owner" }, ], }, }, ann); console.log(`Ann creates the group Customers: ${grouped}`); const [linked, link] = await call("POST", "/invitation-links", { groups: [{ group: "customers" }], label: "the shop's website" }, ann); console.log(`Ann creates an invitation link: ${linked}`); ``` part9.py: ```py ann = sign_up("Ann") opened, shop = call("POST", "/tenants", {"slug": "anns-shop", "name": "Ann's shop"}, ann) ann["tenant"] = shop["tenant"] print(f"Ann opens her shop: {opened}") # Customers may place orders, and work only on orders they own. grouped, _ = call("POST", "/commands/group/group/CreateGroup", { "aggregate_id": "customers", "data": { "name": "Customers", "permissions": [ {"pattern": "shop.Place"}, {"pattern": "shop.AddItem", "when": "owner"}, {"pattern": "shop.Submit", "when": "owner"}, {"pattern": "shop.GetOrder", "when": "owner"}, ], }, }, ann) print(f"Ann creates the group Customers: {grouped}") linked, link = call("POST", "/invitation-links", {"groups": [{"group": "customers"}], "label": "the shop's website"}, ann) print(f"Ann creates an invitation link: {linked}") ``` part9.rs: ```rs let mut ann = client.sign_up("Ann").await?; let open = json!({ "slug": "anns-shop", "name": "Ann's shop" }); let (opened, shop) = client.call("POST", "/tenants", Some(open), &ann).await?; let tenant = shop["tenant"].as_str().map(str::to_owned); ann.tenant = tenant.clone(); println!("Ann opens her shop: {opened}"); // Customers may place orders, and work only on orders they own. let group = json!({ "aggregateId": "customers", "data": { "name": "Customers", "permissions": [ { "pattern": "shop.Place" }, { "pattern": "shop.AddItem", "when": "owner" }, { "pattern": "shop.Submit", "when": "owner" }, { "pattern": "shop.GetOrder", "when": "owner" }, ], }, }); let (grouped, _) = client .call( "POST", "/commands/group/group/CreateGroup", Some(group), &ann, ) .await?; println!("Ann creates the group Customers: {grouped}"); let new_link = json!({ "groups": [{ "group": "customers" }], "label": "the shop's website" }); let (linked, link) = client .call("POST", "/invitation-links", Some(new_link), &ann) .await?; println!("Ann creates an invitation link: {linked}"); ``` Bob and Cleo sign up and join through the link. Bob places an order and works on it; Cleo tries to touch it: part9.ts: ```ts const bob = await signUp("Bob"); const cleo = await signUp("Cleo"); for (const [name, person] of [["Bob", bob], ["Cleo", cleo]] as const) { const [joined] = await call("POST", "/invitation-links/join", { token: link.token }, person); person.tenant = shop.tenant; console.log(`${name} joins through the link: ${joined}`); } const [, me] = await call("GET", "/me", null, bob); const [placed] = await call("POST", "/commands/shop/order/Place", { aggregateId: "o1", data: { customerId: me.senderIdentity } }, bob); console.log(`Bob places o1: ${placed}`); const [added] = await call("POST", "/commands/shop/order/AddItem", { aggregateId: "o1", data: { sku: "apple", qty: 2 } }, bob); console.log(`Bob adds two apples: ${added}`); const [cleoAdds, cleoAddsBody] = await call("POST", "/commands/shop/order/AddItem", { aggregateId: "o1", data: { sku: "pear", qty: 1 } }, cleo); console.log(`Cleo adds to o1: ${cleoAdds} ${cleoAddsBody.error}`); const [bobReads, order] = await call("POST", "/queries/shop/GetOrder", { data: { id: "o1" } }, bob); console.log(`Bob reads o1: ${bobReads} ${order.status}, ${JSON.stringify(order.items)}`); const [cleoReads, refused] = await call("POST", "/queries/shop/GetOrder", { data: { id: "o1" } }, cleo); console.log(`Cleo reads o1: ${cleoReads} ${refused.error}`); ``` part9.py: ```py bob = sign_up("Bob") cleo = sign_up("Cleo") for name, person in (("Bob", bob), ("Cleo", cleo)): joined, _ = call("POST", "/invitation-links/join", {"token": link["token"]}, person) person["tenant"] = shop["tenant"] print(f"{name} joins through the link: {joined}") _, me = call("GET", "/me", None, bob) placed, _ = call("POST", "/commands/shop/order/Place", {"aggregate_id": "o1", "data": {"customerId": me["sender_identity"]}}, bob) print(f"Bob places o1: {placed}") added, _ = call("POST", "/commands/shop/order/AddItem", {"aggregate_id": "o1", "data": {"sku": "apple", "qty": 2}}, bob) print(f"Bob adds two apples: {added}") cleo_adds, cleo_adds_body = call("POST", "/commands/shop/order/AddItem", {"aggregate_id": "o1", "data": {"sku": "pear", "qty": 1}}, cleo) print(f"Cleo adds to o1: {cleo_adds} {cleo_adds_body['error']}") bob_reads, order = call("POST", "/queries/shop/GetOrder", {"data": {"id": "o1"}}, bob) print(f"Bob reads o1: {bob_reads} {order['status']}, {json.dumps(order['items'], separators=(',', ':'))}") cleo_reads, refused = call("POST", "/queries/shop/GetOrder", {"data": {"id": "o1"}}, cleo) print(f"Cleo reads o1: {cleo_reads} {refused['error']}") ``` part9.rs: ```rs let mut bob = client.sign_up("Bob").await?; let mut cleo = client.sign_up("Cleo").await?; for (name, person) in [("Bob", &mut bob), ("Cleo", &mut cleo)] { let join = json!({ "token": link["token"] }); let (joined, _) = client .call("POST", "/invitation-links/join", Some(join), person) .await?; person.tenant = tenant.clone(); println!("{name} joins through the link: {joined}"); } let (_, me) = client.call("GET", "/me", None, &bob).await?; let place = json!({ "aggregateId": "o1", "data": { "customerId": me["senderIdentity"] } }); let (placed, _) = client .call("POST", "/commands/shop/order/Place", Some(place), &bob) .await?; println!("Bob places o1: {placed}"); let add = |sku: &str, qty: i64| json!({ "aggregateId": "o1", "data": { "sku": sku, "qty": qty } }); let (added, _) = client .call( "POST", "/commands/shop/order/AddItem", Some(add("apple", 2)), &bob, ) .await?; println!("Bob adds two apples: {added}"); let (cleo_adds, cleo_adds_body) = client .call( "POST", "/commands/shop/order/AddItem", Some(add("pear", 1)), &cleo, ) .await?; println!( "Cleo adds to o1: {cleo_adds} {}", text(&cleo_adds_body["error"]) ); let get = json!({ "data": { "id": "o1" } }); let (bob_reads, order) = client .call("POST", "/queries/shop/GetOrder", Some(get.clone()), &bob) .await?; println!( "Bob reads o1: {bob_reads} {}, {}", text(&order["status"]), order["items"] ); let (cleo_reads, refused) = client .call("POST", "/queries/shop/GetOrder", Some(get), &cleo) .await?; println!("Cleo reads o1: {cleo_reads} {}", text(&refused["error"])); ``` Bob tries to place an order in Cleo's name: part9.ts: ```ts const [, cleoMe] = await call("GET", "/me", null, cleo); const [forCleo, forCleoBody] = await call("POST", "/commands/shop/order/Place", { aggregateId: "o2", data: { customerId: cleoMe.senderIdentity } }, bob); console.log(`Bob places o2 in Cleo's name: ${forCleo} ${forCleoBody.error}`); ``` part9.py: ```py _, cleo_me = call("GET", "/me", None, cleo) for_cleo, for_cleo_body = call("POST", "/commands/shop/order/Place", {"aggregate_id": "o2", "data": {"customerId": cleo_me["sender_identity"]}}, bob) print(f"Bob places o2 in Cleo's name: {for_cleo} {for_cleo_body['error']}") ``` part9.rs: ```rs let (_, cleo_me) = client.call("GET", "/me", None, &cleo).await?; let for_cleo = json!({ "aggregateId": "o2", "data": { "customerId": cleo_me["senderIdentity"] } }); let (status, body) = client .call("POST", "/commands/shop/order/Place", Some(for_cleo), &bob) .await?; println!( "Bob places o2 in Cleo's name: {status} {}", text(&body["error"]) ); ``` Finally, `Explain` tells Cleo why she may not read the order, and a request without an access token is refused before anything else: part9.ts: ```ts const [, why] = await call("POST", "/queries/group/Explain", { data: { identityId: cleoMe.senderIdentity, type: "shop.GetOrder", aggregateType: "order", aggregateId: "o1" }, }, cleo); console.log(`Why not? allowed: ${why.allowed}, ${why.code}`); const [anonymous, nobody] = await call("POST", "/commands/shop/order/Place", { aggregateId: "o3", data: {} }, { email: "", tenant: shop.tenant }); console.log(`Without signing in: ${anonymous} ${nobody.error}`); ``` part9.py: ```py _, why = call("POST", "/queries/group/Explain", { "data": {"identityId": cleo_me["sender_identity"], "type": "shop.GetOrder", "aggregateType": "order", "aggregateId": "o1"}, }, cleo) print(f"Why not? allowed: {str(why['allowed']).lower()}, {why['code']}") anonymous, nobody = call("POST", "/commands/shop/order/Place", {"aggregate_id": "o3", "data": {}}, {"tenant": shop["tenant"]}) print(f"Without signing in: {anonymous} {nobody['error']}") ``` part9.rs: ```rs let explain = json!({ "data": { "identityId": cleo_me["senderIdentity"], "type": "shop.GetOrder", "aggregateType": "order", "aggregateId": "o1", }, }); let (_, why) = client .call("POST", "/queries/group/Explain", Some(explain), &cleo) .await?; println!( "Why not? allowed: {}, {}", why["allowed"], text(&why["code"]) ); let anonymous = Person { token: None, tenant: tenant.clone(), }; let place = json!({ "aggregateId": "o3", "data": {} }); let (status, nobody) = client .call( "POST", "/commands/shop/order/Place", Some(place), &anonymous, ) .await?; println!("Without signing in: {status} {}", text(&nobody["error"])); ``` TypeScript: ```text Ann signs up and signs in: 200 Ann opens her shop: 201 Ann creates the group Customers: 200 Ann creates an invitation link: 201 Bob signs up and signs in: 200 Cleo signs up and signs in: 200 Bob joins through the link: 200 Cleo joins through the link: 200 Bob places o1: 200 Bob adds two apples: 200 Cleo adds to o1: 403 forbidden Bob reads o1: 200 draft, {"apple":2} Cleo reads o1: 403 forbidden Bob places o2 in Cleo's name: 403 forbidden Why not? allowed: false, forbidden Without signing in: 401 unauthenticated ``` Python: ```text Ann signs up and signs in: 200 Ann opens her shop: 201 Ann creates the group Customers: 200 Ann creates an invitation link: 201 Bob signs up and signs in: 200 Cleo signs up and signs in: 200 Bob joins through the link: 200 Cleo joins through the link: 200 Bob places o1: 200 Bob adds two apples: 200 Cleo adds to o1: 403 forbidden Bob reads o1: 200 draft, {"apple":2} Cleo reads o1: 403 forbidden Bob places o2 in Cleo's name: 403 forbidden Why not? allowed: false, forbidden Without signing in: 401 unauthenticated ``` Rust: ```text Ann signs up and signs in: 200 Ann opens her shop: 201 Ann creates the group Customers: 200 Ann creates an invitation link: 201 Bob signs up and signs in: 200 Cleo signs up and signs in: 200 Bob joins through the link: 200 Cleo joins through the link: 200 Bob places o1: 200 Bob adds two apples: 200 Cleo adds to o1: 403 forbidden Bob reads o1: 200 draft, {"apple":2} Cleo reads o1: 403 forbidden Bob places o2 in Cleo's name: 403 forbidden Why not? allowed: false, forbidden Without signing in: 401 unauthenticated ``` ## What decided - **Who is asking** comes from the access token, checked by the API without the kernel. The middleware still checks that the session is active: signing out ends it at once. - **Ann** is the tenant's owner: owners hold every permission in their tenant. - **Bob** may add to `o1` because he is its owner: the relation `owner` was set from the `OrderPlaced` event. The same holds for reading it, because `GetOrder` declares the order it is about. - **Cleo** is a customer too, but not `o1`'s owner: `forbidden`, for the command and the query. - **Bob's order in Cleo's name** passes the kit (customers may place orders) and is refused by the shop's own middleware, which compares the order's customer with the sender. - Bob and Cleo got the group through the link, without Ann inviting each of them. Ann can limit a link (how many may join, until when) and revoke it at any time. Every sign-in, failed attempt, membership and join is an event in the log. To see the domains in detail, continue with [the kit's documentation](../../kits/). Next: [10. A business process](./10-process). --- # 10. A business process A confirmed order is not done: the customer pays, then the shop ships, and an order nobody pays within three days is cancelled. You could build this from a reactor, a scheduled command and a read model that remembers where each order stands. Here you define it once as a **business process**; the kernel runs it, and its UI shows where every order stands. This part builds on the shop of parts 2 to 7 (the worker, the read models, the reactor and the guard), without the kit of part 9. ## Paying, shipping, cancelling The order gets three commands and their events, in `domain.ts`: domain.ts: ```ts // Part 10: a confirmed order is paid, then shipped; or cancelled when it is not paid in time. orderAggregate .apply("OrderPaid", (s) => { s.status = "paid"; }) .apply("OrderShipped", (s) => { s.status = "shipped"; }) .apply("OrderCancelled", (s) => { s.status = "cancelled"; }) .command("Pay", (s) => { if (s.status !== "confirmed") throw new Rejected(`Cannot pay a ${s.status} order`, "invalid_state"); return [{ type: "OrderPaid" }]; }) .command("Ship", (s) => { if (s.status !== "paid") throw new Rejected(`Cannot ship a ${s.status} order`, "invalid_state"); return [{ type: "OrderShipped" }]; }) .command("Cancel", (s, data: { reason: string }) => { if (s.status !== "confirmed") throw new Rejected(`Cannot cancel a ${s.status} order`, "invalid_state"); return [{ type: "OrderCancelled", data: { reason: data.reason } }]; }); ``` domain.py: ```py # Part 10: a confirmed order is paid, then shipped; or cancelled when it is not paid in time. @order_aggregate.apply("OrderPaid") def _(state, data): state["status"] = "paid" @order_aggregate.apply("OrderShipped") def _(state, data): state["status"] = "shipped" @order_aggregate.apply("OrderCancelled") def _(state, data): state["status"] = "cancelled" @order_aggregate.command("Pay") def _(state, data): if state["status"] != "confirmed": raise Rejected(f"Cannot pay a {state['status']} order", "invalid_state") return [{"type": "OrderPaid"}] @order_aggregate.command("Ship") def _(state, data): if state["status"] != "paid": raise Rejected(f"Cannot ship a {state['status']} order", "invalid_state") return [{"type": "OrderShipped"}] @order_aggregate.command("Cancel") def _(state, data): if state["status"] != "confirmed": raise Rejected(f"Cannot cancel a {state['status']} order", "invalid_state") return [{"type": "OrderCancelled", "data": {"reason": data["reason"]}}] ``` domain.rs: ```rs // Part 10: a confirmed order is paid, then shipped; or cancelled when it is not paid in time. #[derive(Serialize, Deserialize)] struct Cancel { reason: String, } fn fulfil(order_aggregate: &Aggregate) { order_aggregate.apply("OrderPaid", |s, _: Value, _evt, _ctx| { s.status = "paid".into() }); order_aggregate.apply("OrderShipped", |s, _: Value, _evt, _ctx| { s.status = "shipped".into() }); order_aggregate.apply("OrderCancelled", |s, _: Value, _evt, _ctx| { s.status = "cancelled".into() }); order_aggregate.command("Pay", |s, _: Value, _cmd, _ctx| { if s.status != "confirmed" { let message = format!("Cannot pay a {} order", s.status); return Err(Rejected::new(message).code("invalid_state")); } Ok(vec![NewEvent::new("OrderPaid", ())]) }); order_aggregate.command("Ship", |s, _: Value, _cmd, _ctx| { if s.status != "paid" { let message = format!("Cannot ship a {} order", s.status); return Err(Rejected::new(message).code("invalid_state")); } Ok(vec![NewEvent::new("OrderShipped", ())]) }); order_aggregate.command("Cancel", |s, data: Cancel, _cmd, _ctx| { if s.status != "confirmed" { let message = format!("Cannot cancel a {} order", s.status); return Err(Rejected::new(message).code("invalid_state")); } Ok(vec![NewEvent::new("OrderCancelled", data)]) }); } ``` and the orders read model follows them, in `read-models.ts`: read-models.ts: ```ts ordersReadmodel .on("OrderPaid", (_data, evt) => void (orders.get(evt.aggregateId)!.status = "paid")) .on("OrderShipped", (_data, evt) => void (orders.get(evt.aggregateId)!.status = "shipped")) .on("OrderCancelled", (data: { reason: string }, evt) => { const row = orders.get(evt.aggregateId)!; row.status = "cancelled"; row.cancelled = data.reason; }); ``` read_models.py: ```py @orders_readmodel.on("OrderPaid") def _(data, evt): orders[evt.aggregate_id]["status"] = "paid" @orders_readmodel.on("OrderShipped") def _(data, evt): orders[evt.aggregate_id]["status"] = "shipped" @orders_readmodel.on("OrderCancelled") def _(data, evt): orders[evt.aggregate_id].update(status="cancelled", cancelled=data["reason"]) ``` read_models.rs: ```rs for (event_type, status) in [("OrderPaid", "paid"), ("OrderShipped", "shipped")] { let rows = Arc::clone(&orders); orders_readmodel.on(event_type, move |_: Value, evt, _ctx| { rows.lock() .unwrap() .get_mut(&evt.aggregate_id) .expect("placed") .status = status.into(); async { Ok(()) } }); } let rows = Arc::clone(&orders); orders_readmodel.on("OrderCancelled", move |data: Value, evt, _ctx| { let mut rows = rows.lock().unwrap(); let row = rows.get_mut(&evt.aggregate_id).expect("placed"); row.status = "cancelled".into(); row.cancelled = data["reason"].as_str().map(str::to_owned); async { Ok(()) } }); ``` ## The process Also in `domain.ts`, next to the aggregates: domain.ts: ```ts // How long a customer has to pay; PAY_WITHIN=10s to try it. const payWithin = (process.env.PAY_WITHIN ?? "3d") as Duration; export const fulfilment = shopDomain.process("fulfilment") .start("OrderConfirmed", { key: "aggregateId", data: { orderId: "aggregateId" } }) .wait("paid", "OrderPaid", { key: "aggregateId", timeout: payWithin, onTimeout: "cancel" }) .command("ship", "order.Ship", { aggregate: "case.orderId", next: "shipped" }) .command("cancel", "order.Cancel", { aggregate: "case.orderId", data: { reason: value("unpaid") } }) .end("unpaid") .end("shipped"); ``` domain.py: ```py # How long a customer has to pay; PAY_WITHIN=10s to try it. pay_within = os.environ.get("PAY_WITHIN", "3d") fulfilment = ( shop_domain.process("fulfilment") .start("OrderConfirmed", key="aggregate_id", data={"order_id": "aggregate_id"}) .wait("paid", "OrderPaid", key="aggregate_id", timeout=pay_within, on_timeout="cancel") .command("ship", "order.Ship", aggregate="case.order_id", next="shipped") .command("cancel", "order.Cancel", aggregate="case.order_id", data={"reason": value("unpaid")}) .end("unpaid") .end("shipped") ) ``` domain.rs: ```rs fn process(shop_domain: &Domain) -> Process { // How long a customer has to pay; PAY_WITHIN=10s to try it. let pay_within = std::env::var("PAY_WITHIN").unwrap_or_else(|_| "3d".into()); shop_domain .process("fulfilment") .start_named( "OrderConfirmed", Start::key("aggregate_id").data([("order_id", "aggregate_id")]), ) .wait_named( "paid", "OrderPaid", Wait::key("aggregate_id").timeout(&pay_within, "cancel"), ) .command_named( "ship", "order.Ship", Send::aggregate("case.order_id").next("shipped"), ) .command_named( "cancel", "order.Cancel", Send::aggregate("case.order_id").data([("reason", value("unpaid"))]), ) .end("unpaid") .end("shipped") } ``` Read it from the top: - **`start`:** when an order is confirmed (the reactor of part 4 confirms it), a **case** starts for it. The order's ID is the case's key: one running case per order. The case keeps it as `orderId`. - **`wait("paid")`:** the case waits for `OrderPaid` of the same order. If none comes within `PAY_WITHIN` (three days), it goes to `cancel` instead. - **`command("ship")`:** the kernel sends `Ship` to the order and, once it committed, goes on to `shipped`. - **`command("cancel")`:** sends `Cancel` with the reason `unpaid`, then goes on to the next step written, the end `unpaid`. - **`end`:** the two ways a case ends. The UI counts cases by them. Each step has a name: the UI, the statistics and a later version of the process know it by its name. [Business processes](../../concepts/processes) lists every kind of step. ## Let the process act The kernel sends the process's commands as `process:shop.fulfilment`, and the guard of part 6 refuses anyone who is not the order's customer. Tell it what the process may send, in `guard.ts`: guard.ts: ```ts const SERVICES = new Set(["svc-confirmations"]); // What a business process may send (part 10): it acts as `process:`. const PROCESSES: Record = { "process:shop.fulfilment": ["Ship", "Cancel"] }; const registered = new Set(); const ownerOf = new Map(); // order -> customer const guardReadmodel = shopDomain.readModel("guard") .on("CustomerRegistered", (_data, evt) => void registered.add(evt.aggregateId)) .on("CustomerErased", (_data, evt) => void registered.delete(evt.aggregateId)) .on("OrderPlaced", (data: { customerId: string }, evt) => void ownerOf.set(evt.aggregateId, data.customerId)) .middleware({ types: ["shop.*"] }, (check) => { const who = check.request.senderIdentity ?? ""; if (check.type === "Register") return; // anyone may sign up if (check.kind === "query" && SERVICES.has(who)) return; // services may read if (check.kind === "command" && PROCESSES[who]?.includes(check.type)) return; // Files and cache requests carry no data: they have no owner, so only the rules above pass them. const data = (check.data ?? {}) as { id?: string; customerId?: string }; const target = check.kind === "command" ? check.aggregateId! : data.id!; const owner = check.type === "Place" ? data.customerId : check.type === "GetCustomer" || check.aggregateType === "customer" ? target : ownerOf.get(target); if (!registered.has(who) || owner !== who) throw new Rejected(`${check.type} is not allowed for ${who || "anonymous"}`, "forbidden"); }); await new Kernable().listener(guardReadmodel).run(); ``` guard.py: ```py SERVICES = {"svc-confirmations"} # What a business process may send (part 10): it acts as `process:`. PROCESSES = {"process:shop.fulfilment": {"Ship", "Cancel"}} registered: set[str] = set() owner_of: dict[str, str] = {} # order -> customer guard_readmodel = shop_domain.read_model("guard") @guard_readmodel.on("CustomerRegistered") def _(data, evt): registered.add(evt.aggregate_id) @guard_readmodel.on("CustomerErased") def _(data, evt): registered.discard(evt.aggregate_id) @guard_readmodel.on("OrderPlaced") def _(data, evt): owner_of[evt.aggregate_id] = data["customerId"] @guard_readmodel.middleware(["shop.*"]) def _(check: Check): who = check.request.sender_identity if check.type == "Register": return # anyone may sign up if check.kind == "query" and who in SERVICES: return # services may read if check.kind == "command" and check.type in PROCESSES.get(who, ()): return data = check.data or {} # files and cache requests carry none: they have no owner target = check.aggregate_id if check.kind == "command" else data.get("id") if check.type == "Place": owner = check.data.get("customerId") elif check.type == "GetCustomer" or check.aggregate_type == "customer": owner = target else: owner = owner_of.get(target) if who not in registered or owner != who: raise Rejected(f"{check.type} is not allowed for {who or 'anonymous'}", "forbidden") Kernable().listener(guard_readmodel).run() ``` guard.rs: ```rs const SERVICES: [&str; 1] = ["svc-confirmations"]; // What a business process may send (part 10): it acts as `process:`. const PROCESSES: [(&str, &[&str]); 1] = [("process:shop.fulfilment", &["Ship", "Cancel"])]; let registered = Arc::new(Mutex::new(HashSet::::new())); let owner_of = Arc::new(Mutex::new(HashMap::::new())); // order -> customer let shop_domain = shop().domain; let guard_readmodel = shop_domain.read_model("guard"); let r = Arc::clone(®istered); guard_readmodel.on("CustomerRegistered", move |_: Value, evt, _ctx| { r.lock().unwrap().insert(evt.aggregate_id); async { Ok(()) } }); let r = Arc::clone(®istered); guard_readmodel.on("CustomerErased", move |_: Value, evt, _ctx| { r.lock().unwrap().remove(&evt.aggregate_id); async { Ok(()) } }); let o = Arc::clone(&owner_of); guard_readmodel.on("OrderPlaced", move |data: Value, evt, _ctx| { let customer = data["customerId"].as_str().unwrap_or_default().to_owned(); o.lock().unwrap().insert(evt.aggregate_id, customer); async { Ok(()) } }); let (r, o) = (Arc::clone(®istered), Arc::clone(&owner_of)); guard_readmodel.middleware(MiddlewareOptions::new(&["shop.*"]), move |check| { let who = check.request.sender_identity.clone(); let allowed = check.type_name == "Register" // anyone may sign up || (check.kind == CheckKind::Query && SERVICES.contains(&who.as_str())) // services may read || (check.kind == CheckKind::Command && PROCESSES.iter().any(|(process, types)| *process == who && types.contains(&check.type_name.as_str()))) || { let field = |name: &str| check.data[name].as_str().map(str::to_owned); let target = match check.kind { CheckKind::Command | CheckKind::Cancel => check.aggregate_id.clone(), CheckKind::Query => field("id"), // The shop keeps no files: no customer's to protect. CheckKind::Data | CheckKind::Cache => None, }; let owner = if check.type_name == "Place" { field("customerId") } else if check.type_name == "GetCustomer" || check.aggregate_type.as_deref() == Some("customer") { target } else { target.and_then(|t| o.lock().unwrap().get(&t).cloned()) }; r.lock().unwrap().contains(&who) && owner.as_deref() == Some(who.as_str()) }; let who = if who.is_empty() { "anonymous".to_owned() } else { who }; let refusal = format!("{} is not allowed for {who}", check.type_name); async move { if !allowed { return Err(Rejected::new(refusal).code("forbidden").into()); } Ok(()) } }); let kernable = Kernable::new(Options::default())?; kernable.listener(&guard_readmodel).run().await; ``` Without it, the first `Ship` would be refused `forbidden`, and the case would stop there as an **incident**: nothing lost, the case waits at `ship` until you fix the cause and run `kernable processes retry`. ## Register it The kernel runs the process once it is registered, signed with a key that may send what the process sends. An app (`kernable.app`) registers the processes of its domains itself; the shop runs its parts separately, so it registers the process in a script of its own, once per deploy, after the worker is up: fulfilment.ts: ```ts const kernable = new Kernable(); const { name, version, new: changed } = await kernable.registerProcess(fulfilment); console.log(`${name} version ${version}${changed ? "" : " (unchanged)"}`); kernable.close(); ``` fulfilment.py: ```py k = Kernable() registered = k.register_process(fulfilment) print(f"{registered.name} version {registered.version}{'' if registered.new else ' (unchanged)'}") k.close() ``` fulfilment.rs: ```rs let kernable = Kernable::new(Options::default())?; let registered = kernable.register_process(&fulfilment).await?; let unchanged = if registered.new { "" } else { " (unchanged)" }; println!( "{} version {}{unchanged}", registered.name, registered.version ); ``` Restart the worker, the read models and the guard, then register the process with a short time to pay, to see an unpaid order cancelled: ```sh PAY_WITHIN=10s node fulfilment.ts # or .venv/bin/python fulfilment.py, or cargo run --bin fulfilment ``` It prints `shop.fulfilment version 1`. Registering the same process again changes nothing; a changed one becomes version 2, and running cases finish on the version they started with. ## Try it Two orders: Ann pays the first, not the second. part10.ts: ```ts await send("customer", "c1", "Register", { email: "ann@example.com", name: "Ann" }); for (const id of ["o1", "o2"]) { await send("order", id, "Place", { customerId: "c1" }); await send("order", id, "AddItem", { sku: "apple", qty: 1 }); await send("order", id, "Submit"); // confirmed by the reactor: a case starts for the order } await send("order", "o1", "Pay"); console.log("o1:", (await until("o1", "shipped")).status); // o2 is never paid: when its time is up, the process cancels it. const o2 = await until("o2", "cancelled"); console.log("o2:", o2.status, o2.cancelled); ``` part10.py: ```py send("customer", "c1", "Register", {"email": "ann@example.com", "name": "Ann"}) for id in ("o1", "o2"): send("order", id, "Place", {"customerId": "c1"}) send("order", id, "AddItem", {"sku": "apple", "qty": 1}) send("order", id, "Submit") # confirmed by the reactor: a case starts for the order send("order", "o1", "Pay") print("o1:", until("o1", "shipped")["status"]) # o2 is never paid: when its time is up, the process cancels it. o2 = until("o2", "cancelled") print("o2:", o2["status"], o2["cancelled"]) ``` part10.rs: ```rs let register = json!({ "email": "ann@example.com", "name": "Ann" }); send("customer", "c1", "Register", register).await?; for id in ["o1", "o2"] { send("order", id, "Place", json!({ "customerId": "c1" })).await?; send("order", id, "AddItem", json!({ "sku": "apple", "qty": 1 })).await?; send("order", id, "Submit", Value::Null).await?; // confirmed by the reactor: a case starts for the order } send("order", "o1", "Pay", Value::Null).await?; println!("o1: {}", until("o1", "shipped").await?.status); // o2 is never paid: when its time is up, the process cancels it. let o2 = until("o2", "cancelled").await?; println!("o2: {} {}", o2.status, o2.cancelled.unwrap_or_default()); ``` TypeScript: ```text o1: shipped o2: cancelled unpaid ``` Python: ```text o1: shipped o2: cancelled unpaid ``` Rust: ```text o1: shipped o2: cancelled unpaid ``` The script ends after ten seconds: `o2` waited for its payment until its time was up. ## Watch it Open the UI (http://127.0.0.1:7234) and **Processes**: the fulfilment as a diagram, from the confirmed order on the left to its two ends on the right, each path as wide as the orders that took it, with how many are at each step now. Above it: how many ended shipped and unpaid, and how long orders waited for their payment. Click `paid` to list the orders waiting for it, and a case to read its story: what it waited for, what it sent and how it ended, each linked to the order's history. Searching for `o1` in the top bar finds the order and its case. The case's events are in the log like any other, written and signed by the kernel: ```sh kernable processes cases kernable.db shop.fulfilment kernable processes case kernable.db shop.fulfilment o1#1 ``` Next, see [Business processes](../../concepts/processes) for decisions, queries and timers, and what happens on a rejection. --- # Concepts How kernable works and why. Each page is short; the [tutorials](../tutorial/) show the same ideas in code, and the [reference](../reference/) has the details. | Page | Covers | |---|---| | [Architecture](./architecture) | What the kernel does and what your processes do | | [Event sourcing in kernable](./event-sourcing) | Aggregates, commands, events, and how it differs from CRUD | | [Domains](./domains) | What a domain holds, its rules, how it relates to services, a code layout | | [How the pieces fit](./pieces) | Commands, events and queries; aggregates, read models, reactors, middleware and processes, and when you need a process | | [Order and consistency](./ordering) | One command per aggregate, listener order, settling, at-least-once delivery | | [Unique keys and conditions](./uniqueness) | Values unique across aggregates, commands that require a fact of another aggregate | | [Trust](./trust) | Keys, grants, signatures, the hash chain | | [Compliance](./compliance) | The technical controls kernable covers for ISO 27001, SOC 2, the GDPR and others, and where it stops | | [Queries and middleware](./queries-and-middleware) | Why every read goes through the kernel, and how checks fail closed | | [Business processes](./processes) | Flows across commands, events and queries, run by the kernel and shown in its UI | | [Data over time](./data) | Event versions, formats, the vault, snapshots | | [Failures and recovery](./failures) | What happens when something crashes | | [Compared with other systems](./comparison) | kernable next to event stores, frameworks and logs | --- # Architecture kernable splits an application into a **kernel** and **your processes**. **The kernel** (Rust, one process per store) owns everything that must be right for every application: - it checks every signature and every key's grant; - it queues commands, one at a time per aggregate, and hands them to workers; - it commits events atomically, on the version they were based on, into a hash chain; - it delivers events to listeners in a fixed order and waits for them when a submit settles; - it routes queries to the read model that answers them and runs the middleware chain; - it keeps personal data in the vault and snapshots in their own store, and serves files and a shared cache; - it knows each **service's membership**: which workers, listeners and middlewares the service consists of, so it waits for exactly those ([Remove a listener, middleware or aggregate](../how-to/remove-listener)); - it shows all of this live in a browser: its [UI](../how-to/monitor#the-ui) maps services, aggregates, events and listeners, lists the connections, and shows any aggregate's history. **Your processes** own the business logic and nothing else: - **Workers** run aggregates: given the history and a command, they return new events or a rejection. - **Listeners** receive events in order: **read models** keep views and answer queries, **reactors** cause side effects and follow-up commands, a **middleware** checks requests. - **Clients** submit commands and ask queries. Processes talk to the kernel over gRPC: workers and listeners keep one stream open each, clients make single calls. Any language with gRPC can take part; TypeScript, Python and Rust have SDKs. ## A command's way What happens to one command, say `shop.PlaceOrder`: 1. **Sent.** A client signs the command with its own key and submits it. A client is just one of your processes: an API, a script, or a reactor reacting to an event. 2. **Checked.** The kernel verifies the signature and checks that the key's grant allows this command type. Then the middleware chain decides, in order; the first rejection refuses the command, and a refused command is never stored. 3. **Stored and queued.** The kernel stores the command and queues it behind the other commands of the same aggregate: one command per aggregate at a time, so nothing overtakes. Two clients changing the same cart run one after the other; different carts run in parallel. 4. **Executed.** A worker gets the command with the aggregate's history, or a snapshot plus the events after it. It runs your handler and returns new events, or a rejection. It signs its result; it changes nothing itself. 5. **Committed.** The kernel appends the events on the version they were based on, atomically, and extends the hash chain. If another command came first, the worker runs it again on the new history. 6. **Delivered.** Listeners get the events in a fixed order: read models (order 0) first, then reactors (order 10), each acknowledging what it has processed. 7. **Settled.** The submit returns once every listener has acknowledged. So the next query, from any process, already sees the change: there is no polling and no "eventually". The same as a sequence: ```mermaid sequenceDiagram participant C as client participant K as kernel participant M as middleware participant W as worker participant R as read model C->>K: Submit (signed) K->>K: check signature, grant K->>M: check M-->>K: pass K->>K: store command, queue it K->>W: Execute (history, snapshot) W-->>K: Result (events, signed) K->>K: commit, extend the hash chain K->>R: events, in order R-->>K: ack K-->>C: committed and settled ``` ## A query's way What happens to one query, say `shop.GetCart`: 1. **Asked.** A client signs the query with its own key and sends it. Like commands, every query goes through the kernel, also when the read model runs in the same process: a shortcut would be the one place where authorization does not apply. 2. **Checked.** The kernel verifies the signature and checks that the key's grant allows asking this query type. Then the middleware chain runs, exactly as for a command; the first rejection refuses the query. 3. **Routed.** The kernel picks a connected read model that answers `GetCart`. If several do, it spreads the queries over them: that is how a read model scales. With `minPosition` (e.g. the position of a `committed` submit), it waits until the read model has processed that event. 4. **Answered.** The read model gets the query on its stream, in order with its events, so its answer reflects every event it was sent before. It answers from its own state, or rejects (`not_found`). Answers are not signed: they travel on the read model's authenticated stream and are not stored. 5. **Checked after.** A middleware after the query sees the request and the answer, and may refuse it instead, for example a cart that belongs to another customer. 6. **Returned.** The client gets the answer or the rejection. After a settled submit, it already shows the change. If no read model answers within `--query-timeout-ms`, the client gets an error. The same as a sequence: ```mermaid sequenceDiagram participant C as client participant K as kernel participant M as middleware participant R as read model C->>K: Query (signed) K->>K: check signature, grant K->>M: check (before) M-->>K: pass K->>R: query, after the events it has processed R-->>K: answer K->>M: check (after), with the answer M-->>K: pass K-->>C: answer ``` ## What the kernel does not do The kernel never interprets payloads: commands, events and queries are JSON your processes write and read. That is what lets every process be in another language. **Limits.** One kernel serves one store; several kernels can [share the tenants](../how-to/partitions), each tenant on one of them. A kernel restarts in seconds and loses nothing (see [Failures and recovery](./failures)), and a [standby](../how-to/standby) holds every write the primary told anyone of; with a witness, it takes over by itself. --- # Event sourcing in kernable In a CRUD application, a table holds the current state and an update overwrites it. In an event-sourced application, the **events** are stored, never changed, and the state is derived from them. | | CRUD | Event sourcing in kernable | |---|---|---| | What is stored | the current state | every event, in order, forever | | A change | an update | a command, decided into events | | History and audit | added separately, if at all | built in: every event names its command and sender | | Another view of the data | a migration, a query | another read model, built from the events | | Fixing a bug in a view | repair the data | fix the read model, rebuild it | ## Decide and fold Two small functions carry all of it. **Decide**: a command handler looks at the current state and the command, and returns events, or rejects the command. **Fold**: `apply` turns the events, one after another, into the state. The events are what is stored; the state is only ever folded from them. One cart, step by step: | | Command | State before | Decision | Event stored | Version | State after | |---|---|---|---|---|---|---| | 1 | `AddItem` apple × 2 | empty | accept | `ItemAdded {qty: 2}` | 1 | 2 items | | 2 | `AddItem` pear × 1 | 2 items | accept | `ItemAdded {qty: 1}` | 2 | 3 items | | 3 | `CheckOut` | 3 items | accept | `CheckedOut` {orderId} | 3 | 3 items, checked out | | 4 | `AddItem` fig × 1 | checked out | reject: "Cart is checked out" | none | 3 | unchanged | Row 4 changes nothing: a rejection is an answer to the sender, stored with its command, but no event. To know the cart's state at any time, fold its events up to that version; that is also how a worker gets the state for the next command (or from a [snapshot](./data) plus the events after it). ## Aggregates An **aggregate** is the unit of decision: a customer, an order. It has - a **state**, folded from its events by `apply`: it changes the state (a copy) or returns a new one; - **commands**: a handler looks at the state and returns events, or rejects the command with a reason; - a **version**: the number of its events. Every event gets the next version. Aggregates belong to a **domain** (`shop`); types are named `shop.Register` outside their domain. By convention, domain and aggregate names are lowercase and message types (events, commands, queries) capitalized; both cases are valid, and names are case-sensitive. A domain writes only its own events; it learns about other domains through listeners. **Aggregates are short-lived.** None stays in memory between commands: for every command, the worker gets the aggregate's history (or a snapshot and the events after it), folds it into the state, decides, and forgets it again. That is why any worker can take any command, why a crashed worker loses nothing, and why handlers must decide the same way every time they run. ## Streams and the log Every aggregate has its own **stream**, `shop/cart/c1`: its events, numbered by **version** 1, 2, 3, without gaps. All streams together form **one log**, numbered by **global position**, in the order the kernel committed them. - **Within a stream**, the version decides: a command is decided on version 2 and commits as version 3, or not at all if another command came first. One command per aggregate runs at a time, so this rarely happens. - **Across streams**, the position decides: listeners get every event in log order, whatever aggregate it belongs to, so a read model of all carts sees c1 and c2 interleaved exactly as they happened. - A query can wait for a position (`minPosition`), and `kernable verify` checks the log position by position along its hash chain. What that looks like in the store, the `events` table of `kernable.db` (shortened): | global_position | domain | aggregate_type | aggregate_id | version | event_type | payload | |---|---|---|---|---|---|---| | 41 | shop | cart | c1 | 1 | ItemAdded | `{"sku":"apple","qty":2,"price":3}` | | 42 | shop | cart | c2 | 1 | ItemAdded | `{"sku":"fig","qty":1,"price":2}` | | 43 | shop | cart | c1 | 2 | ItemAdded | `{"sku":"pear","qty":1,"price":1}` | | 44 | shop | cart | c1 | 3 | CheckedOut | `{"orderId":"0b9c…","total":7}` | Each row also keeps the command that caused it, its request context, the worker's signature and the hash chain. The same rows, with their commands, are in the kernel's [UI](../how-to/monitor#the-ui) under History. ## Write side and read side Event sourcing in kernable splits an application into two sides, often called CQRS: - **The write side decides.** Aggregates get commands and return events. They see only their own state, never a read model, so a decision depends on nothing but the aggregate's own history. - **The read side is built from events.** Read models fold the events of any aggregates and domains into views that fit their queries: a cart list, a revenue report, a search index. Reactors do the same to cause side effects and follow-up commands. Each side can change on its own: a new view is a new read model rebuilt from the existing events, without a migration of the write side. And because a settled submit returns only once the read side has the events, the two sides never disagree for a reader: [A command's way](./architecture#a-command-s-way) and [A query's way](./architecture#a-query-s-way) show both in motion. ## Handlers Every handler gets the same kinds of arguments, in the same order: the **data** (the payload of the event, command or query), the **envelope** (what arrived: type, IDs, version), and a **context** (in which request, and what the handler may use). | Handler | Arguments | Envelope | Context | |---|---|---|---| | `apply(type, …)` | `state, data, evt, ctx` | `evt`: type, version, domain, aggregate type and ID, tenant, command ID | `ctx`: the request context of the command that caused the event | | `command(type, …)` | `state, data, cmd, ctx` | `cmd`: command ID, type, domain, aggregate type and ID, tenant | `ctx`: the request context, `now()`, `id()`, the deadline | | `query(type, …)` | `data, qry, ctx` | `qry`: query ID, type, domain, tenant | `ctx`: the request context of who asks, the listener's store | | `on(type, …)` | `data, evt, ctx` | `evt`: as for `apply`, plus position, payload version, redacted fields and when the kernel committed it | `ctx`: the request context, `replaying` (committed before this connection: a rebuild, or a reactor's backlog after a restart, so never a reason to skip a reactor's side effects), `follow()`, `submit()` and `query()` for follow-ups, the store | `apply` gets no `now()` and no `id()`: replaying the events must give the same state every time, so a time or an ID the state needs goes into the event's data. ## Rules for handlers - **Deterministic and free of side effects.** A command may run again (after a crash, or after another command changed the aggregate first), and must decide the same way. Use `ctx.now()` and `ctx.id()` instead of the clock and random IDs: they are the same every time the command runs. - **Small aggregates.** All commands of one aggregate run one after another; commands of different aggregates run in parallel. Draw boundaries where decisions must be consistent, not along entities. - **Rejections are results.** A rejected command is stored like a committed one, with its code and message. ## What a submit returns A submit returns the outcome once every listener has processed the events (`settled`, the default), once the events are stored (`committed`), or, with `enqueue`, only the command ID once the command is queued. Every way is idempotent on the command ID. [Order and consistency](./ordering#settling-reading-your-own-writes) compares them. ## Events An event has a **type**, a JSON **payload**, its aggregate's **version**, a **global position** in the whole log, the **request context** of the command that caused it, and a **schema** version of its payload format. See [Data over time](./data) for how events change format without being rewritten. **Naming.** Events say what happened, in the past tense: `ItemAdded`, `OrderPaid`, `CustomerErased`. Commands say what should happen, in the imperative: `AddItem`, `PayOrder`, `EraseCustomer`. Queries ask: `GetCart`, `ListOrders`. Each `domain.Type` exists once; the domain keeps the rest apart (`shop.ItemAdded`, `billing.ItemAdded`). **What events give you**, beyond the current state: - **An audit trail.** Every change names its command, its sender, its key and its time, and the log is verifiable. - **Every view, at any time.** A new read model folds the existing events; nothing has to be migrated first. - **Reactions.** Reactors turn events into follow-up commands, mails or messages to other systems. - **Time travel.** Fold an aggregate up to any version to see its state back then, or a read model up to any position. - **Debugging.** When a state looks wrong, its events show how it came about, step by step. ## Rules across aggregates An aggregate decides alone, on its own history. A rule that spans two aggregates becomes a short conversation between them, carried by a reactor: - **The reactor submits follow-up commands** with `ctx.submit`, so the sender and the correlation of the original request carry over, and with a fixed command ID (`reserve-o-7`), so a redelivered event changes nothing. If a follow-up is rejected, the reactor parks the event and moves on; an operator sees it and retries or dismisses it ([Failures](./failures)). A step whose rejection is part of the conversation (stock ran out) catches it and submits the compensating command instead. - **Each step is a decision of one aggregate**, with its own rejection. Until the last step, the order is in an intermediate state (`placed`, not yet `confirmed`): model it explicitly. - **Uniqueness and conditions are checked by the kernel.** "Every email only once" or "open an account only while its fund is open" need no reactor: an aggregate declares unique keys and facts from its state, a command the facts it requires, and the kernel checks them when the command commits ([Unique keys and conditions](./uniqueness)). ## When it fits Event sourcing pays off when the history matters, or many views read the same facts: - decisions with rules (orders, bookings, approvals, money), where you must know why something is as it is; - audit and compliance: every change names its command, its sender and its key, and the log is verifiable; - several views of the same data, added over time, or integration of other systems through reactors. It costs a little more where none of that matters: a form that only stores what the user typed, or reference data nobody reasons about. Keep such data out of aggregates: in the kernel's [files and cache](../how-to/data-and-cache), or in a plain database next to kernable. ## Glossary | Term | Meaning | |---|---| | **Domain** | A name that scopes aggregates and message types: `shop`, `billing`. Outside it, types are qualified: `shop.ItemAdded`. | | **Aggregate** | The unit of decision: its state, its commands and its events (`shop/cart/c1`). | | **Command** | A request to change one aggregate, signed by its sender: `AddItem`. Decided into events, or rejected. | | **Event** | A fact that happened, stored forever: `ItemAdded`. | | **Stream** | The events of one aggregate, numbered by **version**. | | **Log** | All events of all streams, numbered by **global position**, chained by hash. | | **Query** | A request to read, answered by a read model: `GetCart`. | | **Listener** | A process that gets events in log order: a read model, a reactor, or a middleware. | | **Read model** | A listener that keeps a view and answers queries. | | **Reactor** | A listener that causes side effects and follow-up commands. | | **Middleware** | A listener that checks commands and queries before (or after) the kernel handles them. | | **Worker** | A process that runs aggregates: it decides commands. | | **Service** | Your processes that belong together; its **membership** tells the kernel what to wait for. | | **Settled** | A submit is settled once every listener has processed its events: the next read sees them. | --- # Domains A **domain** is one area of your business: `shop`, `billing`, `identity`. Its name scopes every type in it, so `shop.ItemAdded` and `billing.ItemAdded` never collide, and in code it is one object that holds everything that belongs to the area. ```ts [TypeScript] import { domain } from "@kernable/sdk"; export const shopDomain = domain("shop"); ``` ```python [Python] from kernable import domain shop_domain = domain("shop") ``` ```rust [Rust] use kernable::domain; let shop_domain = domain("shop"); ``` ## What a domain holds Everything you define on the domain object belongs to it: | Defined with | What it is | |---|---| | `aggregate(name, initial)` | an aggregate: its state, its commands, the events it writes ([Event sourcing](./event-sourcing)) | | `readModel(name)` | a read model: a view built from events, answering queries | | `reactor(name)` | a reactor: side effects and follow-up commands | | `.middleware(…)` on a read model | a middleware: checks commands and queries ([Queries and middleware](./queries-and-middleware)) | | `event(type, version)` | an event's payload version, with an `upcast` from each older one ([Data over time](./data)) | | `event(type, …).personal(…)`, `.erases(…)` | which event fields go to the vault, and which event erases a person | | `command(type).personal(…)`, `.secret(…)` | which command fields go to the vault | The names are TypeScript's; Python and Rust spell them `read_model`, and so on ([SDK reference](../reference/)). Types without a domain are the domain's own: on `shopDomain`, `ItemAdded` means `shop.ItemAdded`. Types of other domains are qualified: `billing.InvoicePaid`, or a whole domain, `billing.*`. ## Rules - **A domain writes only its own events.** Its aggregates decide; nothing outside the domain writes `shop.*` events. - **It reacts to other domains through listeners.** A read model of `shop` may read `billing.InvoicePaid`; a reactor of `billing` may react to `shop.CheckedOut` and submit a `billing` command. Each domain stays the only one that decides about its own aggregates. - **Other domains' event versions come with `use`.** A listener that reads another domain's events gets them upcast to the versions it knows with `use(billingDomain)`; its own domain needs no `use`. - **The vault declarations travel with the domain.** A client that submits commands with personal or secret fields needs the domain's declarations, to move those fields out before signing: `kernable.use(shopDomain)`, also when the domain's aggregates run in another process or language. - **Grants follow domains.** A key is granted per type or per domain: `run=shop.*`, `listen=shop.*,billing.InvoicePaid` ([Trust](./trust)). ## Domains, services and apps A domain is a unit of **code and meaning**; a service is a unit of **deployment**: the processes that belong together, with one membership. - **One service, several domains.** An app runs every aggregate, read model and reactor of its domains: `kernable.app({ service: "shop", domains: [shopDomain, billingDomain] })`. - **One domain, one service** is the usual case and the simplest to reason about. - **One domain across services** works too, for example its read models in a separate process. Then run each part with explicit workers and listeners instead of `app()`, and apply each service's membership ([memberships](../how-to/remove-listener)). The membership belongs to the service, not the domain: it lists what the service runs, whichever domains that comes from. ## A layout that works kernable does not prescribe files. One that stays clear as a domain grows: the domain object with its event versions and vault declarations in one file, each aggregate, read model and reactor in its own, and one entry point per service. ``` shop/ domain.ts domain("shop"), event versions, vault declarations cart.ts the cart aggregate order.ts the order aggregate carts.ts a read model mails.ts a reactor billing/ domain.ts invoice.ts invoicing.ts reacts to shop.CheckedOut app.ts kernable.app({ service, version, domains: [shopDomain, billingDomain] }) ``` The same works in Python (`shop/domain.py`, …) and Rust (`src/shop/mod.rs`, …). Each file defines its part on the domain object it imports, so the app only needs the domains. ## Naming - **Domains and aggregates**: lowercase, singular: `shop`, `cart`, `order`. - **Events**: what happened, in the past tense: `ItemAdded`, `OrderPaid`. - **Commands**: what should happen, imperative: `AddItem`, `PayOrder`. - **Queries**: what is asked: `GetCart`, `ListOrders`. - **Read models and reactors**: what they keep or do, lowercase: `carts`, `invoicing`, `mails`. Names are case-sensitive, and the conventions are not enforced: they keep a log that many teams read understandable. --- # How the pieces fit A kernable application is made of a few kinds of code: aggregates, read models, reactors, middleware and processes. You always write aggregates; the rest you add when you need them. **A process is optional**: most commands come straight from your app, and many applications have no process at all. This page shows how the pieces work together, on an example almost every product has: onboarding a new account. ## Three kinds of messages - **A command is a request:** "open account a-17", "activate account a-17". It goes to one aggregate, which may refuse it (`Rejected`, with a code). Your app sends most commands; a reactor or a process may send them too. - **An event is a fact:** "account a-17 was opened". Only the aggregate writes it, when it accepts a command. Events are never changed or deleted, and every reader gets them in order. - **A query is a question:** "what is the state of account a-17?". A read model answers it from the events it has read. ## The pieces, in the order you build them | Piece | Turns | You write it | Tutorial | |---|---|---|---| | **Aggregate** | commands into events: it checks the rules and decides | always: it is the core of every application | [Commands and events](../tutorial/shop/2-commands) | | **Read model** | events into answers to queries | when anything reads: your frontend, your API | [Queries](../tutorial/shop/3-queries) | | **Reactor** | an event into an action: an email, a call to another service, a command | when one event should lead to one thing | [A reactor](../tutorial/shop/4-reactor) | | **Middleware** | nothing: it lets a command or query through, or refuses it | when who may do what depends on more than the key | [Authorization](../tutorial/shop/6-authorization) | | **Process** | events into the next command, over several steps and over time | when a flow waits for things, has deadlines, and you want to see where each case stands | [A business process](../tutorial/shop/10-process) | The aggregate stays the authority over its data. A reactor or a process sends ordinary commands to it, exactly as your app would, and the aggregate checks its rules every time: neither can do anything the aggregate does not allow. ## Do you need a process? | If you think | Write | |---|---| | "When X happens, do Y." (When an account is opened, send a welcome email.) | a **reactor** | | "After X, wait for Y; if it does not come within a deadline, do Z." (After the KYC check is requested, wait for a decision; after 3 days, escalate.) | a **process** | | "Show me how many onboardings are stuck, and where." | a **process**: its cases and statistics are in the UI | A reactor keeps no state between events. A process keeps one **case** per key (here: per account), knows which step it is at, and sets timers. If you notice you are building a read model that remembers "where each account stands" so that a reactor knows what to do next, that is a process. ## An example: onboarding an account The onboarding: a KYC check that a person approves or rejects, the account activated, the email address confirmed, with deadlines and reminders. Each part does one job: | Part | Does | |---|---| | Aggregate `account` | takes `Open`, `RequestKyc`, `DecideKyc`, `Activate`, `SendVerification`, `ConfirmEmail`, `Close`, and writes `AccountOpened`, `KycRequested`, `KycDecided`, `AccountActivated`, … | | Reactor `kyc-provider` | on `KycRequested`, calls the KYC provider; its result (a webhook) comes back into your API | | Your back office | a reviewer approves or rejects: it sends `DecideKyc { approved }` | | Reactor `mailer` | on `VerificationSent`, sends the email with the confirmation link | | Your API | the user clicks the link: it sends `ConfirmEmail` | | **Process `accounts.onboarding`** | says what comes next: sends the commands, waits for the events, keeps the deadlines | The process, defined next to the domain: ```ts [TypeScript] export const onboarding = accounts.process("onboarding") .start("AccountOpened", { key: "aggregateId", data: { accountId: "aggregateId" } }) .command("request-kyc", "account.RequestKyc", { aggregate: "case.accountId" }) // A person decides: the process waits for the event their command writes. .wait("kyc", "KycDecided", { key: "aggregateId", data: { approved: "data.approved" }, timeout: "3d", onTimeout: "kyc-overdue" }) .decide("kyc-result", { when: "case.approved == true", then: "activate", otherwise: "reject" }) .command("activate", "account.Activate", { aggregate: "case.accountId" }) .command("verify-email", "account.SendVerification", { aggregate: "case.accountId" }) .wait("email", "EmailConfirmed", { key: "aggregateId", timeout: "2d", onTimeout: "remind", next: "onboarded" }) .command("remind", "account.SendVerification", { aggregate: "case.accountId" }) .wait("email-late", "EmailConfirmed", { key: "aggregateId", timeout: "7d", onTimeout: "abandoned", next: "onboarded" }) .command("kyc-overdue", "account.EscalateKyc", { aggregate: "case.accountId", next: "kyc" }) .command("reject", "account.Close", { aggregate: "case.accountId", next: "rejected" }) .end("onboarded") .end("rejected") .end("abandoned") .end("closed") // The customer closes the account during onboarding: the case ends at once. .cancelOn("AccountClosed", { key: "aggregateId", end: "closed" }); ``` ```python [Python] onboarding = ( accounts.process("onboarding") .start("AccountOpened", key="aggregate_id", data={"account_id": "aggregate_id"}) .command("request-kyc", "account.RequestKyc", aggregate="case.account_id") # A person decides: the process waits for the event their command writes. .wait("kyc", "KycDecided", key="aggregate_id", data={"approved": "data.approved"}, timeout="3d", on_timeout="kyc-overdue") .decide("kyc-result", when="case.approved == true", then="activate", otherwise="reject") .command("activate", "account.Activate", aggregate="case.account_id") .command("verify-email", "account.SendVerification", aggregate="case.account_id") .wait("email", "EmailConfirmed", key="aggregate_id", timeout="2d", on_timeout="remind", next="onboarded") .command("remind", "account.SendVerification", aggregate="case.account_id") .wait("email-late", "EmailConfirmed", key="aggregate_id", timeout="7d", on_timeout="abandoned", next="onboarded") .command("kyc-overdue", "account.EscalateKyc", aggregate="case.account_id", next="kyc") .command("reject", "account.Close", aggregate="case.account_id", next="rejected") .end("onboarded") .end("rejected") .end("abandoned") .end("closed") # The customer closes the account during onboarding: the case ends at once. .cancel_on("AccountClosed", key="aggregate_id", end="closed") ) ``` ```rust [Rust] let onboarding = accounts .process("onboarding") .start_named("AccountOpened", Start::key("aggregate_id").data([("account_id", "aggregate_id")])) .command_named("request-kyc", "account.RequestKyc", Send::aggregate("case.account_id")) // A person decides: the process waits for the event their command writes. .wait_named("kyc", "KycDecided", Wait::key("aggregate_id").data([("approved", "data.approved")]).timeout("3d", "kyc-overdue")) .decide("kyc-result", Decide::when("case.approved == true", "activate").otherwise("reject")) .command_named("activate", "account.Activate", Send::aggregate("case.account_id")) .command_named("verify-email", "account.SendVerification", Send::aggregate("case.account_id")) .wait_named("email", "EmailConfirmed", Wait::key("aggregate_id").timeout("2d", "remind").next("onboarded")) .command_named("remind", "account.SendVerification", Send::aggregate("case.account_id")) .wait_named("email-late", "EmailConfirmed", Wait::key("aggregate_id").timeout("7d", "abandoned").next("onboarded")) .command_named("kyc-overdue", "account.EscalateKyc", Send::aggregate("case.account_id").next("kyc")) .command_named("reject", "account.Close", Send::aggregate("case.account_id").next("rejected")) .end("onboarded") .end("rejected") .end("abandoned") .end("closed") // The customer closes the account during onboarding: the case ends at once. .cancel_on_named("AccountClosed", "aggregate_id", "closed"); ``` One account, from start to end. Each line is a command (`──Name──▶`) or an event (`▶ Name`): ``` Your API ──Open──────────────▶ account a-17 ▶ AccountOpened Kernel AccountOpened starts the case a-17#1 Process ──RequestKyc────────▶ account a-17 ▶ KycRequested Reactor KycRequested: calls the KYC provider Back office ──DecideKyc─────────▶ account a-17 ▶ KycDecided { approved: true } Process the wait "kyc" completes; decides; ──Activate──▶ account a-17 ▶ AccountActivated Process ──SendVerification──▶ account a-17 ▶ VerificationSent Reactor VerificationSent: sends the email Your API the user clicks the link; ──ConfirmEmail──▶ account a-17 ▶ EmailConfirmed Process the wait "email" completes: the case ends at "onboarded" ``` The process never writes `AccountActivated` itself and never changes the account: it asks, and the account decides. Its own notes, where each case stands, are events of the case (domain `process`), which the UI reads ([Cases in the log](./processes#cases-in-the-log)). ## When something else happens meanwhile A case reacts only to the events it waits for at its current step, with its key. Everything else in the log goes past it. Three situations: - **The awaited event comes early.** The user confirms the email before the case reaches the step `email`. The kernel keeps the event for the case; when the case gets there, the wait completes at once. - **An event the process does not care about.** Support changes the account's address. The case goes on as before. - **An event that should stop the case.** The customer closes the account during onboarding. Name it with `cancelOn`: the case ends at once at the end step you give (`closed`), whatever step it is at. Its timer is withdrawn, a command it still awaits is ignored, and an incident ends with it. Without `cancelOn`, the case would notice only at its next command, which the closed account refuses; give that rejection a path (`rejected: { closed: "closed" }`), or the case stops as an incident. Events the case's own commands caused do not cancel it: the step `reject` sends `Close`, the account writes `AccountClosed`, and the case still ends at `rejected`, where it was going. ## What a process does not do - **Talk to the outside.** Calls to other services, emails and webhooks belong in reactors and your API; their results come back as commands. So the log stays the record of what happened, and a case can be replayed. - **Write your events or change your data.** Only aggregates do, through commands. - **Give people a task list.** Approving a KYC check is your back office's screen; the process waits for the command it sends. A read model, or the cases waiting at `kyc`, give that screen its list. - **Move running cases to a new version.** A changed process is a new version; running cases finish on theirs ([Versions](./processes#registering-versions-and-removing)). Next: [Business processes](./processes) has every step and option; the [tutorial's part 10](../tutorial/shop/10-process) builds one in the shop. --- # Order and consistency kernable gives you a strict order where it matters and parallelism where it does not. ## Commands - **One at a time per aggregate.** The kernel queues commands per aggregate in submission order and leases one at a time to a worker. Commands of different aggregates run in parallel, on as many workers as you start. - **Optimistic commit.** A worker's events are committed only on the version they were based on. If the aggregate changed in between, the kernel runs the command again with the new history. - **Idempotent by ID.** A command ID is stored with the command. Submitting the same ID again returns the original outcome instead of running it twice: give retried commands a fixed ID. - `expected_version` in the request context rejects a command unless the aggregate is still at the version the client saw. ## Listeners Events have a **global position**. Every listener receives them in that order, one at a time, and acknowledges what it has processed. There is **one order for everything**: all domains and all tenants share the global position. A read model that reads `shop` and `billing` events sees them interleaved exactly as they were committed ([a projection across domains](./event-sourcing#write-side-and-read-side)). - **Lower orders first.** A listener gets an event only once every listener with a lower order has acknowledged it. Read models default to 0 and reactors to 10, so a reactor's queries always see the event it reacts to. Listeners of the same order run in parallel. - **At least once.** After a crash a listener gets the events after its last acknowledgement again. Handlers must be idempotent: reactors submit with fixed command IDs, read models overwrite rather than add. A read model with a store writes data and position together and takes effect exactly once. ## Who counts Not every listener holds the others back. Those that **count** decide when higher orders get an event, when a submit settles, and when the kernel is ready (`/ready`): | Listener | Counts | Why | |---|---|---| | registered, in a service without a membership, or without a service | yes | the simple rule: everything registered counts | | listed in its service's membership | yes, also before it ever connected | the service consists of it ([memberships](../how-to/remove-listener)) | | registered, but not listed in its service's membership | no (dynamic) | it runs, keeps its position, nobody waits for it | | live (`live: true`) | no | never registered: starts at the newest event, forgotten when it disconnects | A **middleware** counts like a listener and fails closed besides: while it is away, what it checks is refused. A listener that runs as several **instances** (one per machine, [Run a listener on several machines](../how-to/instances)) counts once: it is through an event once one instance has processed it, and caught up once one instance has, so an instance that is gone or rebuilding holds nothing back. A query goes to an instance that has reached its `min_position`. ## When a listener is slow or stuck A listener that counts and does not acknowledge holds things up, on purpose, rather than letting a reader see an incomplete state: - **Settling submits wait** up to `--settle-timeout-ms` (default 5 s), then throw `NotSettled`. The command **is committed**; only the guarantee that every read model shows it is missing. Retrying with the same command ID returns the same outcome. - **Higher orders wait** behind it: reactors do not get events the read models before them have not processed. - **The kernel is not ready** (`/ready` answers 503 and names the listener), and its UI shows the listener as *behind*, with how many events. - A listener that is away, not slow, is the same case: it counts until its service's membership no longer lists it, or it is removed ([Remove a listener](../how-to/remove-listener)). ## Settling: reading your own writes By default `submit` **settles**: it returns once every listener has processed the command's events. The next query, from anyone, sees the change. No polling, no "eventually". | `wait` | returns when | use it for | |---|---|---| | `settled` (default) | every listener processed the events | APIs that read right after writing | | `committed` | the events are stored | throughput, or when you do not read back | | `accepted` (`enqueue`) | the command is stored in the queue | fire and forget; resubmit the ID to get the outcome | A query can also wait on its own: `minPosition` answers only after the read model has processed that position. **A settled submit includes the reactions it caused.** A reactor (order 10) acknowledges an event only after its handler finished, including the follow-up command it submitted and awaited. So when the original submit settles, the follow-up is committed too, and the read models already show its events. Only reactions the reactor starts without waiting for them come later. ## What is consistent, and from when | | Consistent | How | |---|---|---| | Within one aggregate | always | one command at a time, committed on the version it was decided on | | A read after a settled submit | from the moment the submit returns | every listener that counts has processed the events | | A read after a `committed` submit | from the given position on | ask with `minPosition`, or the read model may not have it yet | | Unique keys and required facts | at every commit | the kernel checks them atomically with the commit ([Unique keys and conditions](./uniqueness)) | | Across aggregates | step by step | a reactor turns one aggregate's event into the next command ([rules across aggregates](./event-sourcing#rules-across-aggregates)); each step is consistent in itself | | Across domains and tenants | in one global order | every listener sees the same sequence of positions | ## A listener never waits for itself A listener handles one event at a time, so a request from inside its handler must not wait for that listener. The SDKs name the listener in such requests, and the kernel adapts: - a settling submit from a listener waits only for listeners of a **lower** order: a reactor still reads its writes in the read models; - a query from a listener goes to another read model, or is refused at once instead of timing out; - a middleware cannot send what it checks itself. --- # Unique keys and conditions An aggregate decides alone, on its own history ([Event sourcing](./event-sourcing#rules-across-aggregates)). Two kinds of rules still reach across aggregates, and the kernel checks both when a command commits: - **Uniqueness.** Every fund has its own ISIN, and no two open funds of a tenant share a name. Each fund decides alone, so two funds launched at the same moment with the same ID would both commit. - **A condition on another aggregate.** An investor account opens only while its fund is open. A check against a read model before the submit leaves a gap: the fund can be wound up between the check and the commit. An aggregate declares **unique keys** and **facts**, both computed from its own state, and a command declares the facts it **requires** of other aggregates. Handlers and events do not change. ## Unique keys A unique key is a name and a function of the state that returns the value the aggregate holds: TypeScript: ```ts export const fundsDomain = domain("funds"); type Fund = { status: "none" | "open" | "wound_up"; isin?: string; name?: string }; export const fundAggregate = fundsDomain.aggregate("fund", { status: "none" }) .apply("FundLaunched", (_s, data: { isin: string; name: string }) => ({ status: "open", ...data })) .apply("FundWoundUp", (s) => { s.status = "wound_up"; }) .command("Launch", (s, data: { isin: string; name: string }) => { if (s.status !== "none") throw new Rejected("Fund already exists", "exists"); return [{ type: "FundLaunched", data: { isin: data.isin, name: data.name } }]; }) .command("WindUp", (s) => { if (s.status !== "open") throw new Rejected(`Cannot wind up a ${s.status} fund`, "invalid_state"); return [{ type: "FundWoundUp" }]; }) // One ISIN per fund in all tenants, kept after winding up. .unique("isin", (s) => s.isin?.trim().toUpperCase(), { scope: "global" }) // One name per open fund in each tenant: winding up frees it. .unique("name", (s) => (s.status === "open" ? s.name?.trim().toLowerCase() : null)) .fact("open", (s) => s.status === "open"); ``` ```python [Python] from kernable import Rejected, Requirement, domain funds_domain = domain("funds") fund_aggregate = funds_domain.aggregate("fund", {"status": "none"}) @fund_aggregate.apply("FundLaunched") def _(state, data): return {"status": "open", **data} @fund_aggregate.apply("FundWoundUp") def _(state, data): state["status"] = "wound_up" @fund_aggregate.command("Launch") def _(state, data): if state["status"] != "none": raise Rejected("Fund already exists", "exists") return [{"type": "FundLaunched", "data": {"isin": data["isin"], "name": data["name"]}}] @fund_aggregate.command("WindUp") def _(state, data): if state["status"] != "open": raise Rejected(f"Cannot wind up a {state['status']} fund", "invalid_state") return [{"type": "FundWoundUp"}] # One ISIN per fund in all tenants, kept after winding up. fund_aggregate.unique("isin", lambda s: s["isin"].strip().upper() if "isin" in s else None, scope="global") # One name per open fund in each tenant: winding up frees it. fund_aggregate.unique("name", lambda s: s["name"].strip().lower() if s["status"] == "open" else None) fund_aggregate.fact("open", lambda s: s["status"] == "open") ``` Rust: ```rust #[derive(Clone, Default, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct Fund { status: String, // "", "open" or "wound_up" isin: Option, name: Option, } #[derive(Deserialize)] #[serde(rename_all = "camelCase")] struct Launch { isin: String, name: String, } fn fund(funds_domain: &Domain) -> Aggregate { let fund_aggregate = funds_domain.aggregate("fund", Fund::default()); fund_aggregate.apply("FundLaunched", |s, data: Launch, _evt, _ctx| { s.status = "open".into(); s.isin = Some(data.isin); s.name = Some(data.name); }); fund_aggregate.apply("FundWoundUp", |s, _: Value, _evt, _ctx| { s.status = "wound_up".into() }); fund_aggregate.command("Launch", |s, data: Launch, _cmd, _ctx| { if !s.status.is_empty() { return Err(Rejected::new("Fund already exists").code("exists")); } Ok(vec![NewEvent::new( "FundLaunched", json!({ "isin": data.isin, "name": data.name }), )]) }); fund_aggregate.command("WindUp", |s, _: Value, _cmd, _ctx| { if s.status != "open" { return Err(Rejected::new("Cannot wind up this fund").code("invalid_state")); } Ok(vec![NewEvent::new("FundWoundUp", json!({}))]) }); // One ISIN per fund in all tenants, kept after winding up. fund_aggregate.unique( "isin", |s| s.isin.as_ref().map(|id| id.trim().to_uppercase()), Unique::global(), ); // One name per open fund in each tenant: winding up frees it. fund_aggregate.unique( "name", |s| { s.name .as_ref() .filter(|_| s.status == "open") .map(|n| n.trim().to_lowercase()) }, Unique::tenant(), ); fund_aggregate.fact("open", |s| s.status == "open"); fund_aggregate } ``` After every command, also one without events, the worker computes the keys from the new state and sends the full set. The kernel compares it with what the aggregate holds and stores the difference with the commit. So an aggregate always holds exactly what its state yields: | The function returns | The aggregate holds | |---|---| | a string | that value | | a list of strings | each value once | | `null`, `undefined`, `None` | nothing | - **Freeing cannot be forgotten.** A changed value frees the old one in the same commit; a value that becomes null frees it. There is no separate step to release a key. - **Scope.** `tenant` (the default) makes a value unique within the command's tenant; another tenant may hold the same value. `global` makes it unique across tenants, also when they live on [different partitions](../how-to/partitions): the control partition reserves each new global key before it is committed. A personal global key is unique within its partition only. - **Per aggregate type.** A key belongs to its domain, aggregate type and name: `isin` of `funds.fund` never collides with a `isin` of another aggregate type. - **Normalising is yours.** The kernel compares the SHA-256 of the value's UTF-8 bytes, so `LU0001` and `lu0001 ` are two values. Trim, lowercase or format the value in the function. - **Deterministic**, like `apply`: the function sees only the state. If it throws, the command is rejected as for a handler error (`handler_error`). ## Facts and requirements A fact is a name and a function of the state that says whether the aggregate holds it: the fund above holds `open` while its status is open. A command declares the facts other aggregates must hold for it to commit: TypeScript: ```ts type Account = { status: "none" | "open"; fundId?: string }; export const accountAggregate = fundsDomain.aggregate("account", { status: "none" }) .apply("AccountOpened", (_s, data: { fundId: string }) => ({ status: "open", fundId: data.fundId })) .command( "OpenAccount", (s, data: { fundId: string }) => { if (s.status !== "none") throw new Rejected("Account already exists", "exists"); return [{ type: "AccountOpened", data: { fundId: data.fundId } }]; }, // Commits only while the fund is open, checked by the kernel at commit. { requires: (data) => [{ domain: "funds", aggregateType: "fund", aggregateId: data.fundId, fact: "open" }] }, ); ``` ```python [Python] account_aggregate = funds_domain.aggregate("account", {"status": "none"}) @account_aggregate.apply("AccountOpened") def _(state, data): return {"status": "open", "fundId": data["fundId"]} # Commits only while the fund is open, checked by the kernel at commit. @account_aggregate.command("OpenAccount", requires=lambda data, state: [Requirement("funds", "fund", data["fundId"], "open")]) def _(state, data): if state["status"] != "none": raise Rejected("Account already exists", "exists") return [{"type": "AccountOpened", "data": {"fundId": data["fundId"]}}] ``` Rust: ```rust #[derive(Clone, Default, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct Account { fund_id: Option, } #[derive(Serialize, Deserialize)] #[serde(rename_all = "camelCase")] struct OpenAccount { fund_id: String, } fn account(funds_domain: &Domain) -> Aggregate { let account_aggregate = funds_domain.aggregate("account", Account::default()); account_aggregate.apply("AccountOpened", |s, data: OpenAccount, _evt, _ctx| { s.fund_id = Some(data.fund_id) }); account_aggregate.command("OpenAccount", |s, data: OpenAccount, _cmd, _ctx| { if s.fund_id.is_some() { return Err(Rejected::new("Account already exists").code("exists")); } Ok(vec![NewEvent::new( "AccountOpened", json!({ "fundId": data.fund_id }), )]) }); // Commits only while the fund is open, checked by the kernel at commit. account_aggregate.requires("OpenAccount", |_s, data: OpenAccount| { [Requirement::new("funds", "fund", &data.fund_id, "open")] }); account_aggregate } ``` - **From the data and the state before the command.** The requirement function gets both; a command that requires a fact of its own aggregate sees what the aggregate holds before it. - **Same tenant.** A requirement names a domain, an aggregate type, an aggregate ID and a fact, always in the command's tenant. The domain may be another one. - **At commit.** The kernel checks what the named aggregate holds at the moment the command commits. If the fund was wound up a moment before, the account is rejected with `requirement_missing`. - **No lock.** A requirement does not hold the fund open afterwards: winding it up later still commits. Blocking a change while other aggregates refer to it is not covered. ### By a key, or absent A requirement can name its aggregate by a **unique key** of its type instead of its ID: the fund whose `isin` is `FND-lyra`. The kernel finds the aggregate that holds the key when the command commits, always in the command's tenant: a global key held in another tenant names no aggregate here, so a requirement never sees another tenant's facts. No aggregate holds it: then none holds the fact either. So a command needs no lookup of the ID first, and the check stays atomic. A requirement can also be **absent**: the aggregate must not hold the fact. An aggregate without events, or a key nobody holds, holds no fact, so "the vehicle is gone or wound up" is the fact `open` absent. Without a domain, a requirement is of the command's own domain. ```ts [TypeScript] requires: (data) => [ // By the fund's ISIN, normalised as its key function does. { domain: "funds", aggregateType: "fund", key: "isin", value: data.fundIsin.toLowerCase(), fact: "open" }, // Only while the vehicle (own domain) is not wound up, or does not exist. { aggregateType: "vehicle", aggregateId: data.vehicleId, fact: "woundUp", absent: true }, ], ``` ```python [Python] requires=lambda data, state: [ Requirement.by_key("fund", "isin", data["fundIsin"].lower(), "open", domain="funds"), Requirement.of("vehicle", data["vehicleId"], "woundUp", absent=True), ] ``` ```rust [Rust] .requires("OpenAccount", |_, d: Open| [ Requirement::by_key("funds", "fund", "isin", &d.fund_isin.to_lowercase(), "open"), Requirement::of("vehicle", &d.vehicle_id, "woundUp").absent(), ]) ``` - **The value as the key has it.** The kernel compares the SHA-256 of the value: normalise it exactly as the aggregate type's key function does (`toLowerCase` above), or nobody holds it. - **Not personal keys.** A personal key lives in the vault and cannot name an aggregate. - **Never the value.** A rejection names the key (`funds.fund with key isin does not hold open`), never its value; `requirement.key` says which, and `requirement.value` is empty. ## Personal keys A key built from personal data, such as an email, names its subject: TypeScript: ```ts type Investor = { investorId?: string; email?: string }; export const investorAggregate = fundsDomain.aggregate("investor", {}) .apply("InvestorRegistered", (_s, data: { investorId: string; email: string }) => ({ ...data })) .command("RegisterInvestor", (s, data: { email: string }, cmd) => { if (s.investorId) throw new Rejected("Investor already registered", "exists"); return [{ type: "InvestorRegistered", data: { investorId: cmd.aggregateId, email: data.email } }]; }) // A personal key: kept in the vault, freed when the investor is erased. .unique("email", (s) => s.email?.trim().toLowerCase(), { personal: (s) => s.investorId }); fundsDomain.command("RegisterInvestor").personal(["email"], (_d, id) => id); fundsDomain.event("InvestorRegistered", 1).personal(["email"], (_d, id) => id); ``` ```python [Python] investor_aggregate = funds_domain.aggregate("investor", {}) @investor_aggregate.apply("InvestorRegistered") def _(state, data): return dict(data) @investor_aggregate.command("RegisterInvestor") def _(state, data, cmd): if "investorId" in state: raise Rejected("Investor already registered", "exists") return [{"type": "InvestorRegistered", "data": {"investorId": cmd.aggregate_id, "email": data["email"]}}] # A personal key: kept in the vault, freed when the investor is erased. investor_aggregate.unique("email", lambda s: s["email"].strip().lower() if s.get("email") else None, personal=lambda s: s.get("investorId")) funds_domain.command("RegisterInvestor").personal(["email"], lambda d, investor_id: investor_id) funds_domain.event("InvestorRegistered", 1).personal(["email"], lambda d, investor_id: investor_id) ``` Rust: ```rust #[derive(Clone, Default, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct Investor { investor_id: Option, email: Option, } #[derive(Deserialize)] struct RegisterInvestor { email: String, } fn investor(funds_domain: &Domain) -> Aggregate { let investor_aggregate = funds_domain.aggregate("investor", Investor::default()); investor_aggregate.apply("InvestorRegistered", |s, data: Investor, _evt, _ctx| { *s = data }); investor_aggregate.command( "RegisterInvestor", |s, data: RegisterInvestor, cmd, _ctx| { if s.investor_id.is_some() { return Err(Rejected::new("Investor already registered").code("exists")); } Ok(vec![NewEvent::new( "InvestorRegistered", json!({ "investorId": cmd.aggregate_id, "email": data.email }), )]) }, ); // A personal key: kept in the vault, freed when the investor is erased. investor_aggregate.unique( "email", |s| s.email.as_ref().map(|e| e.trim().to_lowercase()), Unique::tenant().personal(|s: &Investor| s.investor_id.clone()), ); let aggregate = |_: &Value, id: &str| id.to_owned(); funds_domain .command("RegisterInvestor") .personal(&["email"], aggregate); funds_domain .event("InvestorRegistered", 1) .personal(&["email"], aggregate); investor_aggregate } ``` - **Kept in the vault.** The kernel keeps a personal key in the [vault](./data#personal-data-the-vault), as a value of its subject, never in the store. The kernel needs a vault (`--vault-key`); without one, the command is rejected with `no_vault`. - **Erasing the subject frees it.** After an erasure another aggregate can take the value, and neither the log nor the store keeps a trace of it. - **No subject, no key.** A personal key whose subject is empty or null is not held. - **The worker must see the value.** Declare the field personal on the command and the event, as above, and grant the worker's key `vault_read`: without it the state holds `null`, and the aggregate holds no key. ## What the kernel guarantees The kernel checks keys and requirements while it commits, under the lock that orders commits: - **Never two holders.** Two aggregates never hold the same key at the same time. Of two commands racing for one value, exactly one commits; the other is rejected with `unique_taken`. - **Atomic.** The keys and facts an aggregate holds change with its events, in the same commit. A rejected command writes nothing, and is stored as rejected like any other ([Errors](../reference/errors)). - **Never the value.** A rejection names the key or the fact, not the value, and not the aggregate that holds it. - **Read from the store.** The store files every key by its identity and every aggregate's keys and facts with its stream; a commit looks up what it needs there and the kernel keeps what recently used aggregates hold in memory. So startup and memory do not grow with the number of aggregates, and the kernel never runs your fold code. | Code | Message | Means | |---|---|---| | `unique_taken` | `funds.fund key isin is taken` | Another aggregate holds a key the command would leave its aggregate with. | | `requirement_missing` | `funds.fund f1 does not hold open`, `funds.fund with key isin does not hold open`, `funds.vehicle v1 holds woundUp` | An aggregate the command requires does not hold the fact, or holds one it must not. | | `invalid_index` | says which limit | Too many keys, facts or requirements, or a malformed name: a bug in the aggregate's definition. | A rejection also names what failed as data, so you need not parse the message: `key` is the name of the taken key (`isin`), `requirement` the requirement that does not hold (domain, aggregate type, ID, fact). In TypeScript and Python they are `outcome.key` and `outcome.requirement`, in Rust `Rejected::key` and `Rejected::requirement`. Only the kernel sets them, and they are stored with the rejection, so the same command ID gets them back. - **One reason at a time.** The kernel checks the requirements first, in the order `requires` returns them, then the keys. The rejection names the first requirement that does not hold, or else one taken key. Fix that one and submit again: another requirement or key may fail next. Workers send keys and facts signed over `kernable.v1.index` ([Signatures](../reference/signing#unique-keys-and-facts)). ## Aggregates from before a key An aggregate that had events before a key or fact was declared holds nothing of it until its next command. `reindex` runs a command on each of them now: ```ts [TypeScript] const outcomes = await kernable.reindex({ domain: "funds", aggregateType: "fund", aggregateIds: ["f1", "f2"] }); for (const [id, outcome] of outcomes) if (!outcome.committed) console.log(id, outcome.code, outcome.message); ``` ```python [Python] outcomes = k.reindex("funds", "fund", ["f1", "f2"]) for id, outcome in outcomes.items(): if not outcome.committed: print(id, outcome.code, outcome.message) ``` ```rust [Rust] let outcomes = kernable.reindex(Reindex::new("funds", "fund", ["f1", "f2"])).await?; for (id, outcome) in outcomes { if let Outcome::Rejected(r) = outcome { println!("{id}: {} {}", r.code, r.message); } } ``` - **A built-in command.** Every aggregate type with keys or facts handles `_reindex_` (here `_reindex_fund`). It produces no events; the worker sends the set of the current state, and the kernel stores it. Command types starting with `_reindex_` are reserved for it. - **Grants.** The worker registers the command, so its key needs `run` for `funds._reindex_fund`, which `run=funds.*` covers; the client's key needs `submit` for it ([Keys per service](../how-to/keys)). - **An ID without events** holds nothing, and its reindex commits. - **Duplicates from before.** If two existing funds share a ISIN, the first to run a command takes the key. The second is rejected with `unique_taken`, for every command, until a command changes its value or makes it null. Find duplicates with a read model before you declare a key. ## Limits | Per result (the full set of one aggregate) | At most | |---|---| | unique key values | 64 | | facts | 64 | | requirements | 64 | | a key's or fact's name | 128 bytes, not empty, no dots | The SDKs refuse a name that breaks the rule when the key or fact is declared. A result over a limit is rejected with `invalid_index`. ## What a worker learns A rejection tells the worker's command something about other aggregates: - **Facts of its tenant.** A requirement may name any aggregate of the command's tenant, also in another domain, so a worker learns whether that aggregate holds a fact. - **Values of other tenants.** A global key rejects a value another tenant holds, so a worker learns that the value exists somewhere. It never learns where. Use `tenant` scope unless the value must be unique across tenants. - **The store holds hashes.** The kernel never gets a value, only its SHA-256, and files a key under a hash of tenant (unless global), domain, aggregate type, name and that SHA-256. A short value, such as an email, can still be guessed from its hash by whoever reads the store: keep keys built from personal data personal. ## Testing `index(state)` returns the keys and facts an aggregate holds in a state, without a kernel: `fundAggregate.index(state)` in TypeScript, `fund_aggregate.index(state)` in Python, `fund_aggregate.index(&state)?` in Rust. Use it next to `handle` ([Test your domain](../how-to/test-domain)). That two aggregates cannot hold one key is the kernel's job: test it against a real kernel. --- # Trust kernable assumes that any party may lie, and makes every claim checkable. - **TLS** keeps strangers off the network: over TCP, every process presents a certificate from your CA. - **The signature** proves which key sent a command, a query or a result. - **The grant**, signed by a root key, says what that key may do at all. - **The middleware** decides what the person behind the request may do, from the request context. - **The hash chain** with signed checkpoints proves afterwards that nothing stored was changed: `kernable verify`. ## Processes and people kernable authenticates **processes**, by their keys, not the people using your application. Who the person is, your API says in the request context (`senderIdentity`, `senderTenant`, …), signed with the API's key. The **middleware** decides on it: may this customer see this cart? So two limits apply to every request, and both must allow it: - **The key's grant** limits what the process may do at all, whoever it claims to act for. A compromised API can impersonate its users, but only within its own grant: it cannot run commands it may not submit, or read events it may not listen to. - **The middleware** limits what each person may do. It builds its knowledge from events (who owns which cart, who is blocked) and fails closed while it is away. Signing in the people themselves (passwords, sessions, tokens) happens in your API. [Part 6 of the shop tutorial](../tutorial/shop/6-authorization) shows the pattern; the [kit](../kits/) is a complete version of it as source you copy: accounts with passwords, second factors, passkeys and single sign-on, sessions, tenants, groups, and the middleware that decides on them. ## Keys and grants Every process has an **Ed25519 key** (`kernable keys new`). Its **key ID** is the public key in hex, so a signature can be checked without a registry. A signature proves who sent a message, not that the sender may do it: anyone can create a key. So the kernel trusts **root keys** only (`kernable start --root root.pub`). A root key signs **grants**: a grant says what one key may do, per action and type. ```sh kernable grant root.key app.pub 'submit=shop.*' 'listen=shop.*' # writes app.grant ``` | Action | Covers | |---|---| | `submit` | commands the key may submit | | `run` | commands it may run as a worker; it writes events of their domains only | | `subscribe` | events it may read with a subscription | | `listen` | events it may read as a listener (read model, reactor) | | `query` | queries it may ask | | `answer` | queries it may answer as a read model | | `check` | commands and queries it may check as a middleware | | `privileged` | privileged request context switches it may set (`bypass_authorization`) | | `vault_read` | events and commands whose vault values it gets as a listener or middleware | | `vault_erase` | events that erase a subject it may write as a worker | | `data_read` | domains whose files it may read (`shop.*`) | | `data_write` | domains whose files it may put and delete | | `cache` | domains whose cache it may use | | `manage` | services whose membership it may apply (service names, or `*`); with `*` also `kernable promote` | | `replicate` | `*`: following the kernel as its [standby](../how-to/standby); the key gets every record, file and vault value | | `export` | `*`: [exporting](../how-to/export) the whole kernel; the vault values stay sealed with its vault key | | `backup` | `*`: following the kernel as its [backup follower](../how-to/backup#continuous-backup); the key gets every record and file, the vault values sealed | | `reserved_order` | `*`: registering listeners and middlewares at negative orders, which run before every other listener; for kernable's kits (tenancy, accounts, authorization) | A type is a qualified type (`shop.PlaceOrder`), a whole domain (`shop.*`) or everything (`*`). A bare type (`'shop.*'`) means every action on it. The process sends its grant with every call, in the gRPC metadata entry `kernable-grant`; the SDKs read `.grant` next to the key file. The kernel checks the chain: the grant is signed by a root key, it names the key that signed the request, and neither key is revoked. It keeps the newest grant per key. The kernel's settings hold no keys of services and no permissions, so a new service needs a grant, not a change to the kernel. The root key is used only to issue grants and revocations, so it can stay offline. Several roots (`--root` repeated) allow replacing one. A grant cannot be withdrawn. A newer grant for the same key (by its issue time) replaces it, and the kernel refuses the older one once it has seen the newer one, also after a restart: it stores the newest grant per key. Open streams follow the newer grant at once: a worker, listener or subscription whose key may no longer do what it registered for is disconnected, and one that asked for every event type gets only the types the newer grant allows from then on. Rights a key used before the newer grant stay used; to take rights away from a key that may be stolen, give the service a new key and revoke the old one. ## What is signed | Message | Signed by | Proves | |---|---|---| | command | client | who asked for exactly this, in which context | | result (events, snapshot) | worker | which worker decided, based on which version | | query, subscription | client | who asked; valid 60 s around its timestamp | | stream opening | worker, listener | possession of the key (challenge) | | checkpoint | kernel | the state of the whole log at a position | | grant | root key | what a key may do | | revoke request | root key | that a root key ended a key; valid 60 s around its timestamp | | membership | a key granted `manage` | what a service consists of; valid 60 s around its timestamp | | file and cache request | client | who reads, writes or deletes which file or value; valid 60 s around its timestamp | **Replays** are harmless or refused: a command runs once per command ID, whoever sends it again; a result is bound to its command and the version it was decided on; queries, subscriptions, file and cache requests, revocations and memberships are valid only for 60 s around their timestamp, and mutual TLS keeps others from capturing them. Acknowledgements, query answers and middleware verdicts are not signed: they travel on a stream whose key was proven, and are not stored. The exact byte layouts and test vectors are in [Signatures](../reference/signing). ## The hash chain Every stored event's hash covers its content, its worker's signature, its request context and the previous event's hash. The kernel signs a **checkpoint** of the chain regularly. `kernable verify` recomputes the chain and checks every signature: a changed, removed or reordered event shows, and so does a command without its client's signature. Events after the last checkpoint are protected by the chain only. ## Revoking and changing keys A key that leaked is **revoked** from the time it may have been used: a root key signs the revocation (`kernable revoke`) and sends it to the running kernel. The kernel stores it signed with its own key, refuses the key from then on, ends its sessions, and `kernable verify` lists everything the key signed since that time as suspect. A revocation is never undone. Keys change without a restart: a new key needs only its grant, a listener moves to a new key whose grant `succeeds` the old one, and the kernel reads its own key file again on SIGHUP. See [Keys per service, and changing a key](../how-to/keys). ## If something is compromised | Compromised | What an attacker can do | What limits it, and what you do | |---|---|---| | a client's key | send what its grant allows, as that client, in the name of any person it claims | narrow grants per service; the middleware still decides per person; revoke the key from the suspected time, and `kernable verify` lists everything it signed since as suspect | | a worker's key | connect as a worker for its command types and decide those commands wrongly | it can only answer commands that clients signed, and write events of its own domains; revoke it, and verify flags its results | | a listener's key | read the events, and vault values, its grant allows | narrow `listen` and `vault_read` grants; revoke it | | the kernel's key | sign checkpoints | stored events still carry the client's and worker's signatures, which it cannot forge; replace the key file and reload it (SIGHUP), and verify with every kernel key you had | | a root key | issue any grant | keep root keys offline; with several roots (`--root` repeated), revoke the compromised one and its grants lose their effect | | the store file | change, remove or reorder events | `kernable verify` shows every such change up to the last signed checkpoint; only events after it could be cut off unnoticed, so keep the checkpoint interval short | | the vault key, with the vault file | read personal and secret values | keep both apart from the store and its backups; `kernable vault rekey` re-encrypts with a new key | | the UI's address | read events, payloads and connections | it listens on loopback only, has no login, and never decrypts the vault; reach it through an SSH tunnel | ## Transport Over TCP, the kernel requires **mutual TLS**: every process presents a certificate from your CA. Signatures make a message trustworthy; TLS keeps it private and keeps strangers out. ## Privileged switches The request context is business data and the client decides it, except for **privileged switches** such as `bypass_authorization`, which only keys granted `privileged` for that switch may set. Even then, the middleware decides what a bypass means; the kernel does not skip the chain. --- # Compliance kernable does not make an organization compliant or certified: ISO 27001, SOC 2 and the GDPR cover your whole organization, its risks, policies and people. What kernable does is implement a set of **technical controls** in the software built on it, so you don't build them yourself and can show them to an auditor. This page maps them, and says where kernable stops. The control numbers refer to ISO/IEC 27001:2022 (Annex A), the SOC 2 Trust Services Criteria (2017, revised 2022), PCI DSS 4.0 and the regulations as published. Check them against the version your audit uses. ## What kernable covers | kernable | How you show it | Supports | |---|---|---| | **A complete, tamper-evident history.** Every change is an event; every event is signed by the worker that decided it and hash-chained to the one before; the kernel signs checkpoints of the chain. ([Trust](./trust#the-hash-chain)) | `kernable verify` recomputes the chain and every signature, and reports any changed, removed or reordered event | ISO 27001 A.8.15 logging, A.5.33 protection of records · SOC 2 CC7.2, processing integrity · PCI DSS Req. 10 · HIPAA §164.312(b) audit controls, §164.312(c)(1) integrity · GoBD traceability and immutability · FDA 21 CFR Part 11 §11.10(e) audit trails | | **Who did what, for whom.** Every command carries a request context signed by its client (`senderIdentity`, `onBehalfOf`, correlation and causation IDs), stored with every event it caused. ([Queries and middleware](./queries-and-middleware)) | The history of any aggregate, in the UI or the store | ISO 27001 A.8.15 · SOC 2 CC6.1 · GDPR Art. 5(2) accountability | | **Personal data that can be erased.** Fields marked `personal` go to an encrypted vault, not the log; an erasure is a signed event that deletes a person's values within the tenant and leaves a proof without the person. ([Data over time](./data#personal-data-the-vault)) | The erasing event and its proof; the chain still verifies | GDPR Art. 17 right to erasure, Art. 25 data protection by design, Art. 32 · ISO 27001 A.8.10 information deletion, A.5.34 privacy and protection of PII | | **Access control that fails closed.** Grants signed by a root key limit what each process may do; middleware checks every command, query, file and cache request, and a missing middleware refuses everything it covers. ([Trust](./trust#keys-and-grants), [Queries and middleware](./queries-and-middleware)) | The grants and the signed membership of each service | ISO 27001 A.5.15 access control, A.8.3 information access restriction, A.8.2 privileged access rights · SOC 2 CC6.1, CC6.3 · NIS2 Art. 21(2) | | **Cryptography and key management.** Ed25519 signatures on every message, mutual TLS between processes, AES-256-GCM in the vault; keys rotate and are revoked without a restart, and revocations are stored signed and never undone. ([Trust](./trust#revoking-and-changing-keys), [Mutual TLS](../how-to/tls)) | `kernable verify` lists everything a revoked key signed since its revocation time | ISO 27001 A.8.24 use of cryptography · GDPR Art. 32 · HIPAA §164.312(a)(2)(iv), §164.312(e) | | **Monitoring, backup and recovery.** Metrics and a readiness check, a documented backup and restore, crash tests at every point where the kernel has stored something. ([Monitor](../how-to/monitor), [Back up and restore](../how-to/backup), [Failures](./failures)) | The metrics, your backup runs, the [guarantees](./failures) | ISO 27001 A.8.13 information backup, A.8.16 monitoring activities · DORA Art. 9, Art. 12 | ## Where kernable stops These are yours to cover, with other tools or in your application: - **Reads are not recorded.** The log holds changes. Queries are checked by middleware but not stored; if you must log who read what, do it in an [after-middleware](./queries-and-middleware), or in your API. - **Failover.** A [pair with a witness](../how-to/standby) fails over by itself within seconds, without losing an acknowledged write. Processes given both kernels' addresses turn to the new primary by themselves; availability requirements (SOC 2 availability, DORA, NIS2) still need a runbook for the witness and the machines, and backups. - **The event store is not encrypted as a whole.** Only vault values are. Encrypt the disk or volume the store lives on. - **The request context is stored in clear** and is not erased with a person. Use IDs that do not identify a person by themselves ([What stays](../how-to/erasure#what-stays)). - **Backups keep erased data.** Restoring an older vault brings erased values back until you delete them again ([Backups and erasure](../how-to/backup#backups-and-erasure)). - **Events after the last checkpoint** are protected by the chain only. Keep `--checkpoint-interval-ms` short. - **People, not processes.** kernable authenticates processes by their keys. Signing in people (passwords, SSO, sessions) is your application's part. - **Retention periods.** Events are never deleted, and kernable has no automatic retention: personal data leaves through erasures, which your application triggers when a period ends. - **The organization.** Risk assessment, policies, training, suppliers, incident handling: no software covers these. --- # Queries and middleware ## Every read goes through the kernel A read model's state is private to its process. Every other process reads it with a **query**, sent to the kernel like a command: - it is signed, and the key's grant must allow it (`query` to ask, `answer` to answer); - it passes the middleware chain, exactly like a command; - the kernel routes it to a read model that answers its type, **in order with that read model's events**: the answer reflects every event the read model was sent before the query. There are no shortcuts, also not inside one process. A shortcut would be the one place where authorization does not apply. If several read models answer the same type, the kernel spreads queries over them: that is how a read model scales. ## Designing queries - **One query type per question**: `GetCart`, `ListOpenOrders`, `GetRevenue`. A read model may answer several; each type is answered by read models that keep the data for it. - **Unknown is a rejection.** Throw `Rejected` with a code (`not_found`): the client gets it as a result, not as an error. - **After a `committed` submit, ask with `minPosition`**, the position of its events: the answer waits until the read model has them. After a settled submit, it is not needed. - **A listener cannot ask its own read model**: the answer would wait behind the event it is handling. The kernel refuses it at once; ask another read model, or keep what you need in the listener itself. - **No read model connected?** The query waits up to `--query-timeout-ms` (default 5 s) for one, then fails: run read models next to the clients that need them, or several of them. - **A query about one aggregate says so.** `GetOrder { id: "o-17" }` is about order `o-17`, but only your code knows that `id` names an order. Declare it once on the query, `static target = { aggregateType: "order", field: "id" }` on a query class, or `shopDomain.query("GetOrder").target("order", "id")` for a query given by name. The SDK then sends the aggregate with the query, signed, and a middleware sees it as `check.target`, the same as a command's aggregate. That is what lets a rule say "only the order's owner may read it". ## Middleware A **middleware** is a listener that checks commands and queries before the kernel accepts or routes them (or after, see below). It builds its knowledge from events, like a read model, and answers each check from it. It checks **file and cache requests** too, before they run, under their own types: `.PutData`, `.GetData`, `.DeleteData`, `.SetCache`, `.GetCache` and `.DeleteCache`. A middleware that covers `shop.*` or `*` gets them; one that covers `AddItem` does not. It sees who asks (the request context), the tenant, the domain and the key, never the file or the value itself. A refusal reaches the client as `PERMISSION_DENIED`, with the middleware's code. - **Registered in code.** A listener declares the types it checks and its order; its key needs a grant for `check` on those types. Negative orders are reserved for kernable's [kits](./trust#keys-and-grants): they need a grant of `reserved_order`. If its service has a membership, it must list the listener as a middleware ([membership](../how-to/remove-listener)). - **A chain.** Middlewares run by order, then by name; the first rejection refuses the request. A refused command is never stored. - **Fails closed.** The kernel keeps the registration. While a middleware is away, also right after a kernel restart, every request it covers is refused with `UNAVAILABLE`. Only a membership of its service that no longer lists it as a middleware, or `kernable listeners remove`, unregisters it. - **Never stale.** After a reconnect, a middleware is asked only once it has processed every event committed before; checks travel in order with its events. - **Checks only.** A middleware decides; it cannot change a request, because the request is signed. A whole middleware: it learns who is a customer from events, and refuses shop requests of anyone else. TypeScript: ```ts export const identityDomain = domain("identity"); const known = new Set(); const blocked = new Set(); // A read model that is also a middleware: it learns from events, and decides on every shop request. export const rbacReadmodel = identityDomain.readModel("rbac") .on("CustomerRegistered", (_data, evt) => void known.add(evt.aggregateId)) .on("CustomerBlocked", (_data, evt) => void blocked.add(evt.aggregateId)) .middleware({ types: ["shop.*"], order: 0 }, (check) => { const who = check.request.senderIdentity; if (!who) throw new Rejected("who are you?", "unauthenticated"); if (!known.has(who)) throw new Rejected(`${who} is no customer`, "unknown_customer"); if (blocked.has(who)) throw new Rejected(`${who} is blocked`, "blocked"); }); ``` ```python [Python] identity_domain = domain("identity") rbac_readmodel = identity_domain.read_model("rbac") known, blocked = set(), set() @rbac_readmodel.on("CustomerRegistered") def _(data, evt): known.add(evt.aggregate_id) @rbac_readmodel.on("CustomerBlocked") def _(data, evt): blocked.add(evt.aggregate_id) @rbac_readmodel.middleware(["shop.*"], order=0) def _(check): who = check.request.sender_identity if not who: raise Rejected("who are you?", "unauthenticated") if who not in known: raise Rejected(f"{who} is no customer", "unknown_customer") if who in blocked: raise Rejected(f"{who} is blocked", "blocked") ``` Rust: ```rust let identity_domain = domain("identity"); let known = Arc::new(Mutex::new(HashSet::new())); let blocked = Arc::new(Mutex::new(HashSet::new())); let (k, b) = (Arc::clone(&known), Arc::clone(&blocked)); let rbac_readmodel = identity_domain .read_model("rbac") .on("CustomerRegistered", move |_: Value, evt, _ctx| { k.lock().unwrap().insert(evt.aggregate_id); async { Ok(()) } }) .on("CustomerBlocked", move |_: Value, evt, _ctx| { b.lock().unwrap().insert(evt.aggregate_id); async { Ok(()) } }); let (k, b) = (Arc::clone(&known), Arc::clone(&blocked)); rbac_readmodel.middleware(MiddlewareOptions::new(&["shop.*"]), move |check| { let who = check.request.sender_identity.clone(); let refusal = if who.is_empty() { Some(Rejected::new("who are you?").code("unauthenticated")) } else if !k.lock().unwrap().contains(&who) { Some(Rejected::new(format!("{who} is no customer")).code("unknown_customer")) } else if b.lock().unwrap().contains(&who) { Some(Rejected::new(format!("{who} is blocked")).code("blocked")) } else { None }; async move { refusal.map_or(Ok(()), |r| Err(r.into())) } }); ``` Run it like any read model; its key needs `listen` for the identity events and `check` for `shop.*`. ## What middleware is for | Middleware | Runs | Decides on | |---|---|---| | authorization (rbac) | before | the sender, and who owns what, from events | | tenant isolation | before | `senderTenant` against the request's tenant, for commands, queries, files and cache values | | quota, rate limit | before | counts from events, per sender or per aggregate | | maintenance mode, feature flag | before | a switch, kept as an event | | privacy filter | after a query | the answer against the asker | | audit of refusals | after a command | rejections, which never become events | ## What it costs Every request a middleware covers waits for it: one round trip per middleware in the chain, in order. So keep the decision in memory, from what the middleware learned from events, with no calls to databases or other services. A middleware that does not answer within `--middleware-timeout-ms` (default 5 s) refuses the request; the kernel's metrics (`kernable_middleware_seconds`, passed and rejected counts) show what the chain costs. Cover only the types that need a check: `shop.*` rather than `*`. ## Middleware after a request A middleware can also run **after** a request (`when: "after"`); a listener has at most one before and one after. After a request, the handler gets the request and what came of it: | Request | Outcome | What the middleware can do | |---|---|---| | query, answered | the response | refuse it: the client gets the rejection instead. Fails closed, like before. | | query, rejected | code and message | observe | | command | committed with its events, or rejected (also by a middleware before) | observe | Observing is **best effort**: the kernel tells the middleware once the command completes, also for `wait: "accepted"` and for commands that finish after a kernel restart, but nobody waits for it, and what happens while it is away is missed. For anything that must not be missed, use a reactor: events are delivered at least once. Middleware after a request gets no vault values, and like before it changes nothing. Typical uses: an audit trail of refused requests, which never become events; keeping a query from returning data the asker must not see. ## The request context Every command and query carries a signed **request context**. The kernel stores it with every event the command causes, and middleware decides on it. `follow()` gives a follow-up command the context of the event that caused it. | Field | Means | |---|---| | `senderIdentity` | the person or system the request is for: who middleware checks | | `senderTenant`, `senderAccount`, `senderSession` | the sender's tenant, account and session, as your application defines them | | `onBehalfOf` | act as this identity of the target tenant; the sender stays the real principal | | `correlationId` | shared by everything one original request causes; defaults to the command ID | | `causationId` | the command that caused this one (`follow()` sets it) | | `traceId` | your tracing system's ID | | `expectedVersion` | reject the command (`version_mismatch`) unless the aggregate is still at this version | | `bypassAuthorization` | privileged: only keys granted `privileged` for it may set it; what it means, middleware decides | | `attributes` | free key-value pairs for business data, never for security decisions | --- # Business processes A business process is a flow across commands, events and queries: *a confirmed order is paid within 3 days and shipped, else cancelled.* Without help, such a flow ends up spread over reactors, scheduled commands and a read model that remembers where each order stands, and nobody can see it. In kernable you define it once, in code next to your domain; the **kernel runs it**, keeps every step in the log, and its UI shows the diagram, where each case stands, and statistics. A process is optional, and it builds on the other pieces: it sends ordinary commands to your aggregates and waits for the events they write; it never changes your data itself. "When X, do Y" is a reactor; a process is for flows that wait, have deadlines, and whose cases you want to see. [How the pieces fit](./pieces) shows where it belongs, on an account's onboarding. ## A process The shop's fulfilment, from [part 10 of the tutorial](../tutorial/shop/10-process): domain.ts: ```ts // How long a customer has to pay; PAY_WITHIN=10s to try it. const payWithin = (process.env.PAY_WITHIN ?? "3d") as Duration; export const fulfilment = shopDomain.process("fulfilment") .start("OrderConfirmed", { key: "aggregateId", data: { orderId: "aggregateId" } }) .wait("paid", "OrderPaid", { key: "aggregateId", timeout: payWithin, onTimeout: "cancel" }) .command("ship", "order.Ship", { aggregate: "case.orderId", next: "shipped" }) .command("cancel", "order.Cancel", { aggregate: "case.orderId", data: { reason: value("unpaid") } }) .end("unpaid") .end("shipped"); ``` domain.py: ```py # How long a customer has to pay; PAY_WITHIN=10s to try it. pay_within = os.environ.get("PAY_WITHIN", "3d") fulfilment = ( shop_domain.process("fulfilment") .start("OrderConfirmed", key="aggregate_id", data={"order_id": "aggregate_id"}) .wait("paid", "OrderPaid", key="aggregate_id", timeout=pay_within, on_timeout="cancel") .command("ship", "order.Ship", aggregate="case.order_id", next="shipped") .command("cancel", "order.Cancel", aggregate="case.order_id", data={"reason": value("unpaid")}) .end("unpaid") .end("shipped") ) ``` domain.rs: ```rs fn process(shop_domain: &Domain) -> Process { // How long a customer has to pay; PAY_WITHIN=10s to try it. let pay_within = std::env::var("PAY_WITHIN").unwrap_or_else(|_| "3d".into()); shop_domain .process("fulfilment") .start_named( "OrderConfirmed", Start::key("aggregate_id").data([("order_id", "aggregate_id")]), ) .wait_named( "paid", "OrderPaid", Wait::key("aggregate_id").timeout(&pay_within, "cancel"), ) .command_named( "ship", "order.Ship", Send::aggregate("case.order_id").next("shipped"), ) .command_named( "cancel", "order.Cancel", Send::aggregate("case.order_id").data([("reason", value("unpaid"))]), ) .end("unpaid") .end("shipped") } ``` The SDK turns this into a **model**, plain JSON, and the kernel runs it. One engine, in the kernel, runs the processes of every language; your processes only define and register them. | Step | Does | Options | |---|---|---| | `start(Event, …)` | starts a case when the event commits | `key`: where the case's key is; `data`: the case's data from the event | | `command(name, Command, …)` | sends the command | `aggregate`, `data`, `rejected` (code → step), `next` | | `wait(name, Event, …)` | waits for an event of this case | `key`, `data` (merged into the case's data), `timeout` with `onTimeout`, `next` | | `timer(name, …)` | waits for a time | `after` (a duration) or `at` (a field of the case holding a time in ms), `next` | | `decide(name, …)` | picks a path by the case's data | `when` with `then`, or a list of them; `otherwise` | | `query(name, Query, …)` | asks a read model; its answer picks the path | `data`, `answer`, `paths`, `into`, `otherwise` | | `end(name)` | ends the case | | | `cancelOn(Event, …)` | ends a running case at once when the event comes, at any step (not a step itself) | `key`, `end`: an `end` step | - **Steps run in the order written**, unless a step names `next`. Step names are required: the UI, the statistics and new versions know a step by its name. - **References** say where a value comes from: `aggregateId` and `data.` of the event being handled (in `start` and `wait`), `case.` of the case's data, and in a query step `answer` or `answer.`. Python and Rust write fields in their own style (`case.order_id`); the SDK converts them. A literal is `value(…)`. - **Conditions** (`decide`) are small expressions on the case's data: references, numbers, strings, `true`, `false`, `null`, `== != < <= > >=`, `and`, `or`, `not` and parentheses: `case.total > 1000 and not case.trusted`. Anything more is a query to a read model you write. - **Durations** are `30s`, `15m`, `12h` or `3d`. - **Names:** an event or query without a domain is of the process's domain; a command is `aggregate.Command` (`order.Ship`) or `domain.aggregate.Command`. Classes work too; in Rust the typed steps take the type (`command::`) and `start_named`, `wait_named`, `command_named` and `query_named` take a name. No other logic runs in the kernel: what a step does is a command, and what decides is the case's data or a read model's answer. ## How a case runs - **It starts** when its start event commits: one case per key (here the order). While a case for a key runs, another start event for that key is ignored; after it ended, the next one starts a new case. Cases are numbered per key: `o1#1`, then `o1#2`. - **A command step** sends its command and waits for the outcome. Its ID is fixed by the case and the step, so a crash or a failover never sends it twice. Committed, the case goes on. Rejected, it goes to the step `rejected` names for that code; any other rejection stops the case as an [incident](#incidents). - **A wait** completes with the first event of its type whose key is the case's. An event that comes before the case reaches its wait (a fast payment, a slow confirmation) is kept for the case and counts when it gets there. - **A cancelling event** (`cancelOn`) of the case's key ends it at once at the end step named, whatever step it is at: its timer is withdrawn, a command or query it still awaits is ignored when it comes, an incident ends with it. Events the case's own commands caused (their correlation is the case) do not cancel it, so a step that closes the account does not cut its own case short. - **A timeout or a timer** is a [scheduled command](../how-to/schedule) of the case, cancelled when the awaited event comes first. It survives restarts, failover and moves like any other. - **A query step** asks with the position of the event that moved the case, so the read model has seen it ([reading your own writes](./ordering)). While no read model answers (one reconnecting after a restart, say), it asks again every second, for up to 30 s, before the case stops as an incident `query_failed`. - **Tenants:** a case lives in the tenant of the event that started it, and its commands, queries and waits stay in that tenant. With [partitions](../how-to/partitions), each partition runs the cases of its own tenants, and a tenant's running cases move with it. ## Cases in the log Each case is an aggregate of its own: domain `process`, type the process's name, ID the case's. Its events say what happened, in order: | Event | When | |---|---| | `ProcessStarted` | the case started: the version, the key, the case's data, the event that started it | | `StepReached` | the case reached a step | | `CommandSent` | a command step sent its command (with its ID) | | `StepCompleted` | a step completed: how (`committed`, `event`, `timeout`, `answer`, a rejection code, `skipped`), and the step next | | `TimerSet`, `TimerFired` | a timeout or a timer was set, and fell due | | `EventKept` | an event a later wait of the case may need came early | | `IncidentRaised`, `IncidentResolved` | the case stopped, and an operator resolved it | | `CaseCancelled` | an event named in `cancelOn` ended the case: the step it was at, the event and its position, the end | | `ProcessEnded` | the case ended, at which end | The kernel writes and signs them itself; `kernable verify` refuses case events signed by any other key. Because they are ordinary events, backups, the standby, exports and the UI's Explore cover cases like everything else, and you can read them like any other events. ## As whom a process acts The engine's commands and queries carry the request context `senderIdentity: "process:shop.fulfilment"`, with the case in `correlationId` (`shop.fulfilment/o1#1`) and the `process` and `case` attributes. Your middleware decides on them as on any sender: in the tutorial, the guard lets `process:shop.fulfilment` send `Ship` and `Cancel`. With the [kit](../kits/identity#processes), each process gets an identity and an API key of its own, with permissions through groups like a service. ## Registering, versions and removing `kernable.app({ domains })` registers the processes of its domains once its workers are connected; `kernable.registerProcess(fulfilment)` does it on its own, for example in a deploy script. The key needs a grant `process` for the process, and `submit` for each command, `query` for each query and `listen` for each event its model names: a process never does more than the key that registered it. The kernel checks the model when it is registered (every step reachable, every target a step, every condition well-formed, every command type known), so a typo fails the deploy and names the step, not a case later. - **Versions:** registering the same model again changes nothing. A changed one becomes the next version: new cases start on it, **running cases finish on the version they started with**. - **Removing:** a process the service's membership no longer lists, or one removed with `kernable processes remove`, starts no new cases; its running cases finish. Registering it again makes it active. ## Incidents When a case cannot go on, it stops as an **incident**: a rejection with no path for its code, a query that fails or has no path for its answer, a missing field the step needs, or decisions that loop ([the codes](../reference/errors)). Nothing is lost: the case waits at its step. Fix the cause, then resolve it, signed with an operator's key granted `process` for the process: ```sh kernable processes retry shop.fulfilment o1#1 # run the step again kernable processes skip shop.fulfilment o1#1 # go on to the step's next step kernable processes cancel shop.fulfilment o1#1 # end the case ``` The resolution is an event of the case (`IncidentResolved`, with the key that signed it). ## Watching processes - **The UI's Processes** draw each version as a diagram (left to right, or top to bottom when it is long), each path as wide as the cases that took it, with the cases at each step now, incidents in red, and the conditions of its decisions; show outcomes over time and how long open cases have waited; list the cases and tell one, at its own address, as a story linked to the aggregates it touched, with its incident and the commands that resolve it first; over the last hour, day or week ([Monitor kernable](../how-to/monitor#the-ui)). Explore's **Catalog** lists every process with what it starts on, waits for, sends and asks, and its open cases and incidents. - **Metrics:** `kernable_process_cases` by process and state, `kernable_process_incidents` and `kernable_process_step_seconds` per step ([Metrics](../reference/metrics)). - **The command line:** `kernable processes list`, `show`, `cases` and `case` read the store, also with the kernel stopped ([CLI](../reference/cli)). ## Not yet Processes are edited in code only; the UI shows them. There are no human tasks ("wait for a person's decision" is a wait for the event their command writes), and running cases are not moved to a new version. --- # Data over time The log is kept forever, so everything about its data is designed to change without rewriting it. ## Where kernable keeps what | Data | Kept | Deletable | Verifiable | Who reads it | |---|---|---|---|---| | **events** (the log) | forever | never | hash chain, worker signatures | listeners and subscriptions, per grant | | **commands** | forever, with their outcome | never | client signatures | the kernel; `kernable verify` | | **vault values** | until the person is erased; secrets until their command completes | by an erasing event | references in the log | keys granted `vault_read` | | **snapshots** | until replaced, erased, or of an old `schema` | yes: a cache | worker signature | workers | | **files** | until deleted | yes, but **not** by an erasure | signed requests | keys granted `data_read` | | **cache** values | until they expire or the kernel restarts | yes | signed requests | keys granted `cache` | | **read model stores** | in your process | yours to manage | no | the read model | The log only grows: there is no compaction, on purpose, because a verifiable history cannot lose events. Events are small JSON, so a store of millions of events is usually a few gigabytes. Keep data that should expire in the cache, large binaries in files, and back up the store and the vault ([Back up and restore](../how-to/backup)). ## Event versions Every event carries its payload's **schema** version (from 1), signed and part of the hash. To change a payload, raise the version and add an **upcast** from the old one: ```ts export const shopDomain = domain("shop"); shopDomain.event("ItemAdded", 2).upcast(1, (d: { sku: string; qty: number }) => ({ sku: d.sku, count: d.qty })); ``` Aggregates and listeners get every event upcast to the version they know; stored events stay as they were: | Position | Stored as | Schema | A reader on version 2 gets | |---|---|---|---| | 41 | `{"sku":"apple","qty":2}` | 1 | `{"sku":"apple","count":2}`, upcast | | 57 | `{"sku":"pear","count":3}` | 2 | `{"sku":"pear","count":3}` | **When to raise the version:** whenever a reader on the old version would misread the new payload, by renaming, removing, or changing the meaning or type of a field. A new optional field needs no new version if every reader can ignore it; it does if old readers must not miss it. Upcasts chain: a version-1 payload reaches version 3 through `upcast(1, …)` and `upcast(2, …)`. Rules: - **A process that meets a newer version stops** instead of misreading it: it logs, retries, and keeps its position until it is updated. Listeners after it wait too, because of the order. So **deploy readers before writers**. - **A worker cannot write an older version** than the store holds: the kernel refuses it. Old and new workers never write side by side. - A handler marked `raw` gets events as stored, for archives and exports. ## Formats and protocol - **Protocol versions.** Every request carries the protocol version of its SDK; the kernel accepts a range and names the side to update otherwise. - **Store formats.** A store records its data format. A kernel opens only its own format and refuses others at startup. A frozen store of the current format is kept in kernable's tests and must stay readable. - **Before the first release** the protocol and every format stay at version 1 and change in place, without migrations. Use SDKs and a kernel of the same build, and start a new store after updating a development build: an older one may be refused or read wrongly. ## Personal data: the vault Events cannot be deleted, but personal data must be deletable. Fields declared `personal` (with the person they belong to) or `secret` go to the **vault** instead of the log; the log keeps a salted reference: ```json { "name": { "$vault": "41a4ffe62b20…" }, "email": { "$vault": "c56522f357b1…" } } ``` - Workers and listeners granted `vault_read` get the values; everyone else gets `null`. - Secrets are deleted once their command completes. - An **erasing event** deletes a person's values in every stream and domain of its tenant, and the snapshots that may contain them. Read models must forget the person when they see it; rebuilt, they read `null`. - The vault keeps each value encrypted with the vault key, together with its subject and whether it is secret. An erasure leaves a **proof** without the person: the SHA-256 of the subject, the erasing command, how many values went, and when. - **The log records every erasure that happened.** The kernel stores the erasing event first and erases right after, before it answers; after a crash in between it finishes the erasure when it starts, before it serves anything. An erasing command that is rejected erases nothing. - **Files and cache values are not erased** with the person: they are not part of the vault. Delete a person's files yourself when you erase them ([Erase a person](../how-to/erasure)). ## Snapshots and read model stores - **Snapshots** of aggregates live in their own store, not in the log: a cache that workers send every 100 events (`snapshotEvery`) and rebuild whenever it is gone or of another `schema`. By default the kernel keeps them in SQLite next to the event store, so they survive a restart. Without a usable snapshot, a worker gets all of an aggregate's events, in several messages if they do not fit one. The kernel accepts a snapshot only from the worker that ran the command, signed by it, and only at exactly the version after the command's events; it stores it with the hash of that event. A worker that cannot read a snapshot (an unknown `schema`) rejects it, and the kernel drops it and sends all events instead. - **Read models** can keep their state in a **store** in their own process: data and position are written together per batch, so a restart continues where it left off. A new `schema` rebuilds from the first event. --- # Failures and recovery kernable is built so that a crash anywhere loses nothing and does nothing twice. kernable's tests abort the real kernel at each point where it has stored something but not yet told anyone, restart it, and check exactly that. | When this fails | what happens | what you do | |---|---|---| | A **worker** crashes during a command | The kernel hands the command to another worker at once (or after `--lease-ttl-ms` if the connection hangs). It runs again from the same history, so it decides the same way. | nothing; keep handlers deterministic | | A **listener** crashes | It reconnects and gets the events after its last acknowledgement again. Submits that settle wait for it (up to `--settle-timeout-ms`). | make handlers idempotent, or use a store | | A **command handler throws** (a bug, not `Rejected`) | The command is rejected with the code `handler_error`. The rejection is kept for good and goes back to the sender, so it names only the error's type (`TypeError`; Rust: `handler failed`): error texts can hold personal data or internals. The worker logs the whole error. It is not retried: a deterministic handler would throw again. No event is stored. | fix the bug; the sender sees the code and can submit again, with a new command ID (the same ID returns the stored rejection) | | A **read model's handler throws** on one event | The read model reconnects and gets that event again, until the handler gets past it: skipping it would leave its state wrong. It and every listener of a higher order wait; settling submits end in `NotSettled`. | fix the handler and deploy it; the [UI](../how-to/monitor#the-ui) shows the listener as *behind* | | A **reactor's follow-up is rejected** (`ctx.submit`), or its handler throws `Rejected` | The reactor **parks** the event at once and moves on: the kernel keeps its position, type, code and message for the listener. Retrying the same command ID returns the same rejection, so trying again would never get further. | `kernable listeners parked` lists it, the UI and `kernable_listener_parked` show it; fix the cause (or the handler), then `kernable listeners retry` it, or `dismiss` it | | A **reactor's handler throws** something else (a bug, a mail server that is down) | It gets the event again, backing off like a reconnect; after 5 attempts (`parkAfter`, `park_after`) it parks the event as `handler_error`, with only the error's type, and moves on. The log has the whole error. | as above; set `parkAfter: 0` to keep waiting instead. An event that **erases** a subject is never parked: the reactor tries until its handler gets past it, so no personal data stays behind | | A reactor's handler fails because the **kernel is away or busy** (`UNAVAILABLE`, `DEADLINE_EXCEEDED`, `RESOURCE_EXHAUSTED`, `NotSettled`) or its store fails | It waits and tries again for as long as it takes; this never parks. | bring the kernel back | | A listener's **store** cannot be opened (a full disk) | The listener logs an error and tries again, backing off like a reconnect (up to 30 s), until the store opens; the rest of the app keeps running. Until then it and every listener of a higher order wait. | free space or fix the path | | A submit does not **settle** in time | `NotSettled` after `--settle-timeout-ms`. The command is committed; only the guarantee that every listener has it is missing. | find the listener that is behind; submitting the same ID again returns the same outcome | | No worker **registered** an aggregate | Its commands are refused with `NOT_FOUND` at once, also a typo in a type. Once registered, its commands wait in the queue while its workers are away, and `/ready` answers 503. | start the worker before sending its commands | | A **middleware** is away | Everything it checks is refused with `UNAVAILABLE` until it is back. | run it next to the kernel; deploy scripts wait for `/ready` | | The **kernel** crashes | Every stored command is still queued and runs right after the restart; listeners resume after their last acknowledgement. Clients get a connection error. | resubmit with the same command ID: you get the original outcome | | The **network** between a process and the kernel drops | A worker's commands go to another worker, at once or when its lease expires; a listener resumes after its last acknowledgement. A client's request waits for the connection up to its timeout (60 s by default), then fails. The SDKs reconnect by themselves, waiting longer after each failed attempt (up to 30 s); keepalive pings notice a connection that died silently within 40 s. | resubmit with the same command ID | | A process's **clock** is off by more than 60 s | Its queries, subscriptions, file and cache requests and memberships are refused (`expired; check the clock`). Commands carry no time window and are not affected. | keep clocks in sync (NTP) | | A **key is revoked** | The kernel refuses it from then on, ends its connections, and rejects its commands still in the queue. | give the service a new key and grant | | The kernel gets **SIGTERM** | It refuses new requests, lets running ones finish for `--drain-timeout-ms`, stores its state and exits cleanly. The primary of a pair hands its role to the standby in sync first, so the standby serves at once. | stop the kernel this way for deploys | | The **file log** is torn by a crash mid-write | The torn last record is moved aside and cut off on the next start. | nothing | | The **machine** loses power or its OS crashes | With `--durability full` (the default) every committed command was on the disk before anyone was told: nothing acknowledged is lost. With `normal` (the default of `--dev`) the last moments of commands may be lost. | keep `full` in production | | The **disk** is full or fails a write | The request fails with `INTERNAL` and nothing half-written stays behind; a write that failed after it took effect is found and answered with its stored result. A command whose events cannot be stored waits in the queue and runs again after a pause that doubles each time, up to 5 s, until the disk takes it. If the file log cannot sync, it refuses every further write until a restart. | free space or fix the disk; resubmit with the same command ID. A file write that failed is signed anew (the same signature counts as tried: it may have taken effect) | | A **second kernel** opens the same store | It is refused: the file log and SQLite (`.lock`) admit one writer. `kernable verify` and the inspection commands still read a running kernel's store. | run one kernel per store | | The **primary's machine** is lost (with a [standby](../how-to/standby)) | The standby holds every write anyone was told of: commits, listener acknowledgements, files. Writes the primary had not confirmed yet are gone, and nobody heard of them. With a witness, the standby takes over by itself within about 4 s. | point your processes at the new primary; without a witness, `kernable promote` it first | | The **standby** is away (`--replication sync`) | With a witness, the primary commits alone after `--standby-timeout-ms`. Without one, its writes wait: commands stay queued, submits time out, `/ready` answers 503. Back, the standby catches up. | bring it back; without a witness, or restart the primary without `--replication sync` (see [Run a standby](../how-to/standby)) | | A file is **damaged or changed** | Damage in the middle refuses to open; a consistent rewrite shows in `kernable verify`. | restore from a backup | | A process speaks an **old protocol** or meets a **newer event version** | It is refused with a message, or stops at that event. | update it | ## What kernable guarantees - A command is stored before it runs and completes **at most once**: committed or rejected, with all its events or none. - With `--durability full`, what a client, worker or listener was told is on the disk: a power loss does not take it back. - Events are never lost, reordered or duplicated in the log; `kernable verify` proves it. - Listeners see every event, in order, **at least once**; with a store, exactly once. - A settled submit has been processed by every listener that [counts](./ordering#who-counts). The tests prove it, with both stores, in three ways, and check the same invariants every time (no event lost, reordered or duplicated, one outcome per command, every listener given every event it acknowledged, an intact hash chain): - they abort the kernel at each of the points below, restart it, and check that nothing was lost or done twice; - they `kill -9` it at random moments while clients keep placing orders, and restart it, hundreds of times; - they run it on a store that fails a share of its reads and writes, before or after the write took effect. | Crash point | Stored | Not yet | |---|---|---| | `submit.enqueued` | the command | the client's answer | | `execute.sent` | nothing for it yet | a worker has the command | | `commit.stored` | the events | anyone was told | | `snapshot.stored` | the snapshot, after the events | anyone was told | | `reject.stored` | the rejection | anyone was told | | `listeners.acked` | a listener's acknowledgement, in memory | its position on disk | | `listeners.flushed` | listener positions | — | | `settle.done` | every listener processed the events | the client's answer | | `checkpoint.stored` | a signed checkpoint | — | | `vault.stored` | a client's vault values | the command in the queue | | `erasure.intended` | the subject's erasure, noted | the erasing event | | `erasure.pending` | the erasing event | the subject's values erased (done at the next start, before anything is served) | | `erasure.done` | the erasing event, the subject's values erased | anyone was told | | `membership.stored` | a service's membership | the membership in effect | | `revocation.stored` | a revocation | the key refused | ## What it does not - **One standby.** A pair has one primary and one standby; a second standby comes later. - Side effects of reactors (emails, calls to other systems) happen at least once. Make them idempotent, or record them as commands first. ## Finding the cause | What you see | Where to look | |---|---| | submits end in `NotSettled` | the UI's Overview and Operations: which listener is *behind*, and by how many events; `/ready` names it | | commands wait and never run | the UI's Explore › Domains or Catalog: an aggregate without a worker (red); `kernable_queued_commands` in the metrics | | requests refused with `UNAVAILABLE` | a middleware is away: the UI's Overview names it as missing | | commands refused with `NOT_FOUND` | no worker registered the aggregate or command type, and no membership lists it: often a typo | | commands rejected with `handler_error` | the rejection names the error's type; the worker's log has the whole error, with its stack | | a listener restarts again and again | its log: `connection lost (…)` with the handler's error, then `connected again at position N`; it is stuck at the event after N | --- # Compared with other systems kernable is an **application kernel**: storage, ordering, trust and delivery in one process, with business logic in yours. Other tools cover parts of that. This page places kernable among them; it describes their general approach, not every feature of every version. | | kernable | A database, an outbox and a message bus (built yourself) | Event store databases (e.g. EventStoreDB/KurrentDB) | Event sourcing frameworks (e.g. Axon) | Logs and streams (e.g. Kafka) | |---|---|---|---|---|---| | What it is | kernel and runtime for event-sourced applications | the usual setup: tables, plus events published from them | a database for event streams | a library and server for one platform | a distributed log for messages | | Business logic runs | in your processes, any language, driven by the kernel | in your services | in your application, which reads and appends | in your application, on the framework's platform | in your consumers and producers | | Languages | any with gRPC; SDKs for TypeScript, Python and Rust share one API | any | clients for several languages | mostly one platform (e.g. the JVM) | clients for many languages | | One decision per aggregate at a time | yes, the kernel queues commands per aggregate | row locks or optimistic versions, per table, yours | with optimistic concurrency on append; queuing is yours | yes, within the framework | per partition key, if you design for it; concurrency control is yours | | Read your own writes | built in: a submit settles after every listener | across services, you build it | you build it, e.g. by waiting on subscriptions | depends on the setup | you build it | | Order across listeners | fixed: lower orders first | per consumer, yours | each subscription on its own | depends on the setup | each consumer group on its own | | History of every change | the log itself, with command and sender | audit tables, if you add them | the streams | the event store | as long as the topic retains it | | Signed requests and results, verifiable hash chain | built in | not built in | not the database's concern | not built in | not built in | | Authorization of every command and query | middleware in the kernel, fails closed | in each service | access control on streams | in your application | access control on topics | | Personal data that must be erasable | vault with erasing events | delete rows; copies in events and backups are yours | you design it (e.g. crypto-shredding) | you design it | you design it (e.g. compaction, encryption) | | Changing an event's format | versions with upcasts; old readers stop instead of misreading | migrations, and versioned messages | yours | upcasters | schema registry, if you add one | | Delivery to your processes | the kernel delivers to every connected process; no separate broker | the message bus, a separate server | subscriptions from the database | an event bus, often a separate server | it is the broker | | Files and shared cache | included: signed requests, governed by grants | separate systems | separate systems | separate systems | separate systems | | What a service consists of | memberships: the kernel waits for exactly what is listed (`/ready`) | your deploy tooling | your deploy tooling | your deploy tooling | your deploy tooling | | Seeing what runs | a built-in UI: map, connections, any aggregate's history; metrics | dashboards you assemble | a database UI | depends on the setup | tools around the cluster | | Operating it | one binary and a SQLite file (or a log file) | a database and a message bus | a database server or cluster | a server or cluster, plus your application | a cluster | | Scale | one kernel per tenant's partition, with a standby; tenants spread over partitions | as far as you build it | clusters | clusters | clusters, very large | | License | MIT | — | open source and commercial editions | open source and commercial editions | open source, and commercial offerings | **Choose kernable** when the logic of one application is spread over teams or languages, when you need to prove what happened and who asked for it, and when "write, then read it back" must simply work. **Choose something else** when you need a multi-node cluster today, or a message bus between many independent systems rather than one application's kernel. kernable can sit next to such systems: reactors can publish to them. ## Where kernable is young - **One kernel per store.** A kernel restarts in seconds and loses nothing, and a [pair with a witness](../how-to/standby) fails over by itself; but every write goes through the one primary. - **No compaction.** The log only grows; that is what keeps it verifiable. Expiring data belongs in the cache. - **SQLite sets the limits of one store**: a single writer on one machine. Commands of different aggregates run in parallel on the workers, but every commit goes through that one writer. - **Three SDKs.** Other languages speak the gRPC protocol directly ([Write an SDK](../how-to/new-sdk)). - **A young project.** The data format and the protocol stay at version 1 and change in place until the first release, without migrations, and the API may still change before 1.0. ## Moving an existing application You do not have to rewrite anything at once: 1. **Start with one aggregate where decisions matter**: orders, bookings, payments. Its commands go through kernable; everything else stays. 2. **Keep your tables as read models.** A read model, or a reactor, writes the events into the tables your existing code reads, so the rest of the application does not notice. 3. **Tell other systems through reactors.** A reactor publishes events to your message bus, or calls the systems that need them. 4. **Move the next aggregate** when the first one runs. The old and the new parts run side by side as long as needed. --- # How-to guides Short guides for tasks you meet once your application runs. They assume you know the [concepts](../concepts/); the [tutorial](../tutorial/shop/) shows the code they refer to. ## Build and change your application | Guide | When you need it | |---|---| | [Test your domain](./test-domain) | You want to test aggregates without a kernel, or against a real one. | | [Add tenants, accounts and authorization](../kits/getting-started) | Your application needs tenants, people who sign in, roles and invitations. | | [Schedule a command](./schedule) | A command should run later: a timeout, a reminder, a deadline. | | [Change an event's format](./event-versions) | An event must carry something new, but stored events stay as they are. | | [Rebuild a read model](./rebuild-read-model) | You fixed a handler bug, or a read model should hold something new. | | [Remove a listener, middleware or aggregate](./remove-listener) | You stop running something, and the kernel should stop waiting for it. Covers memberships. | ## Run it in production | Guide | When you need it | |---|---| | [Deploy and upgrade](./deploy) | You start, stop or upgrade the kernel and your processes. | | [Back up and restore](./backup) | You set up backups, or you need to restore. | | [Move a kernel with export and import](./export) | You leave a hosted kernel, move to another machine, or change the store backend. | | [Share tenants across kernels](./partitions) | One kernel is not enough for all your tenants: several share them by a signed map. | | [Run a standby](./standby) | A second kernel should hold every write and take over, with a witness by itself, when the primary's machine is lost. | | [Run a listener on several machines](./instances) | The same read models and reactors run on more than one machine, each with its own copy. | | [Run a node from one file](./node) | One file says what runs on a machine: the kernel and its backup follower, started again when they exit or the file changes. | | [Run kernable on your own servers](./self-host) | From a bare Linux machine to a node that starts with it: install, a user, keys, the node's file, systemd, the firewall. | | [Monitor kernable](./monitor) | You want metrics, health checks, alerts and logs, or a reactor parked events. | ## Security | Guide | When you need it | |---|---| | [Keys per service, and changing a key](./keys) | You give each service its own key and grant, or you change, revoke or replace a key. | | [Mutual TLS with your own PKI](./tls) | You run over TCP with your own CA instead of development certificates. | | [Passwords and other secrets](./secrets) | A command carries a value that must never be stored. | ## Data | Guide | When you need it | |---|---| | [Handle an erasure request](./erasure) | A person asks you to erase their personal data. | | [Store files and cache values](./data-and-cache) | A domain needs files (an invoice PDF) or values that expire (a rate limit). | ## Extend kernable | Guide | When you need it | |---|---| | [Write an SDK for another language](./new-sdk) | You want processes in a language without an SDK. | --- # Deploy and upgrade Use this page when you put a new version of a service into production, or upgrade the kernel itself. Both can happen while the other side keeps running: processes reconnect on their own, and the kernel keeps commands queued until a worker takes them. Before you start, give the kernel a metrics address (`--metrics-listen 127.0.0.1:9233`): `GET /ready` there tells you when a deploy is done ([Monitor kernable](./monitor#health-checks)). ## A new version of a service 1. **Issue grants for anything new.** A new service, or a service that needs more rights, gets a new grant from the root key ([Keys per service](./keys)). The kernel does not change. 2. **Deploy readers before writers** when an event's format changes: listeners that read the new format go first, workers that write it after ([Change an event's format](./event-versions)). 3. **Apply the service's membership, once per deploy.** It lists the workers (aggregate types), listeners and middlewares the kernel waits for. From a pipeline, with a key granted `manage=`: ```sh kernable apply 127.0.0.1:7233 keys/deploy.key shop.json # shop.json: {"service": "shop", "workers": ["shop.cart"], "listeners": ["carts", "mails"], # "middlewares": ["guard"], "metadata": {"version": "1.4.0"}} ``` It answers `applied shop: 1 workers, 2 listeners, 1 middlewares`. From code, call `kernable.apply(...)` in a deploy script; an app (`kernable.app({ service, version, domains }).run()`) builds the membership from its domains and applies it itself on every start, so its key needs `manage` for the service. 4. **Start the new processes and stop the old ones.** Workers and listeners can restart at any time: a worker's commands go to another worker, a listener continues after its last acknowledged event. 5. **Wait for `/ready`.** It answers 200 once every listed worker and listener is connected and the listeners have caught up; until then 503 names what it waits for: ```sh $ curl -s 127.0.0.1:9233/ready waiting for listeners: carts $ curl -s 127.0.0.1:9233/ready ready ``` 6. **Check the result.** `kernable services ` shows each applied membership, who applied it and when; a listed member that never connected is marked `(never registered)`. The kernel UI shows the same live ([Monitor kernable](./monitor#the-ui)). What the new membership no longer lists stops counting: the kernel stops waiting for it, and a middleware that is no longer listed stops checking ([Remove a listener, middleware or aggregate](./remove-listener)). ### Pitfalls - **The newest apply wins.** The kernel refuses a membership issued before the one it holds, so a late or replayed apply cannot bring an old list back. Apply from one place per deploy, not from every process. With `app()`, every start applies: an old replica that restarts during a rolling deploy brings the old list back until a new one starts. Finish the rollout, or apply from the pipeline afterwards. - **One service is one membership.** Two different apps with the same service name replace each other's list. - **Commands of unknown types are refused.** A worker registers its aggregate and command types when it connects; until then, commands for an aggregate or command type that no worker registered and the membership does not list get `NOT_FOUND`. Start the worker of a new aggregate before clients send its commands, or list it in the membership: commands of a listed aggregate wait in the queue until its worker comes. - **Middleware is always checked.** While a listed middleware restarts, the requests it checks are refused. Keep the restart short, or run a second instance under a different name. ## The kernel A [pair with a witness](./standby#a-pair-with-a-witness) updates without downtime: [update the standby, switch over, update the old primary](./standby#update-a-pair-without-downtime). A single kernel: 1. **Back up** the store and the vault ([Back up and restore](./backup)). 2. **Stop the kernel with SIGTERM** (or SIGINT). It refuses new submits and queries, lets running ones finish for `--drain-timeout-ms` (default 10 s), then signs a checkpoint, saves listener positions and exits. Commands that did not finish stay queued and run again after the restart. 3. **Install the new `kernable` binary.** 4. **Start the kernel** with the same flags, and wait until `GET /ready` answers 200. Before the first release a store of an earlier development build does not carry over: formats change in place, without migrations. While the kernel is away, workers and listeners reconnect on their own (their console shows `connection lost (…), reconnecting`, then `connected again`), waiting a little longer after each failed attempt, up to 30 s. A client's requests wait for the kernel up to their timeout (60 s by default); a request that gives up gets `DEADLINE_EXCEEDED`, and should be resubmitted with the same command ID: a command ID runs once. **SDK versions.** Every SDK sends its protocol version. A kernel accepts a range and refuses others with a message saying whether the SDK or the kernel needs an update. Upgrade the kernel first when a release says so. Before the first release, deploy SDKs and kernel of the same build together. ## What needs a restart | Change | What to do | |---|---| | a new service, new rights | issue a grant; no change to the kernel | | a leaked key | `kernable revoke` against the running kernel ([Keys](./keys#revoking-a-stolen-key)) | | the kernel's key | replace the file and send SIGHUP ([Keys](./keys#the-kernel-s-key)) | | TLS certificates, the vault key, any other flag of `kernable start` | restart the kernel ([kernable start](../reference/configuration)) | --- # Back up and restore Read this before you run a kernel with data you cannot lose. The event store is the system of record: everything else either depends on it (the vault, the keys) or is rebuilt from it (snapshots, read models). To copy a whole kernel into one file, for another machine or a self-hosted kernel, use [export and import](./export). ## What to back up | What | Flag (default) | Why | If lost | |---|---|---|---| | the event store | `--store` (`sqlite:kernable.db`) | commands, events, checkpoints, revocations, registrations of listeners and workers, applied memberships | everything is lost | | the vault | `--vault-store` (`sqlite:vault.db`) | personal and secret values | personal data reads as `null` everywhere | | the vault key | `--vault-key` | decrypts the vault | the vault cannot be read: same as losing it | | files | `--data-store` (`file:data`) | files your processes put ([Store files](./data-and-cache)) | the files are gone | | the kernel key | `--kernel-key` (`kernel.key`) | signs checkpoints and revocations | change it ([Keys](./keys#the-kernel-s-key)); keep every old public key for `kernable verify` | | the root key | `--root` takes its `.pub` | signs grants and revocations | start with a new root and grant every key again ([Keys](./keys#the-root-key)) | | process keys and `.grant` files | — | sign requests; say what each key may do | create new keys and grant them with the root key | | snapshots | `--snapshots` (`sqlite:kernable-snapshots.db`, next to the event store) | a cache | nothing: workers rebuild them, the first command per aggregate reads all its events | | the cache | `--cache-store` (`memory:`) | values that expire | nothing: it is in memory and empty after every restart | | read model stores | in your processes | caches of your views | nothing: read models rebuild from the events | With `kernable start --dev`, all of these are in the working directory: `kernable.db`, `kernable-snapshots.db`, `vault.db`, `data/`, and the keys `root`, `kernel`, `vault` and `app` (`.key` and `.pub`), with `app.grant`. Where each kind of data lives and how long it is kept: [Data over time](../concepts/data#where-kernable-keeps-what). ## Back up 1. **Back up the event store, then the vault**, right after each other. In that order, the worst case is a vault value that no event refers to yet, which does no harm. With SQLite, use SQLite's online backup; it is safe while the kernel runs: ```sh sqlite3 kernable.db ".backup backup/kernable.db" sqlite3 vault.db ".backup backup/vault.db" ``` Do not `cp` a running SQLite store: the event store runs in WAL mode, and the latest commits are still in `kernable.db-wal`. To copy plain files, stop the kernel first. A `file:` event log can be copied while the kernel runs. The copy may end in a torn record; the kernel cuts it off when it opens the copy and keeps the cut bytes in `.torn-`. 2. **Copy the data directory** (`data/` by default). Each file is written next to its place and then renamed, so a copy never holds half a file. 3. **Check the backup** of the event store: ```sh kernable verify sqlite:backup/kernable.db kernel.pub ``` Name every public key the kernel ever had. It works on a copy, without a running kernel: ```text 2311 events, 1903 commands, 41 checkpoints, 0 events after the last checkpoint OK ``` `0 events` means the path is wrong: `verify` opens a missing SQLite file as an empty store. `OK, 1 suspect` (exit code 2) means the log is intact but something was signed by a revoked key; `TAMPERED` (exit code 1) means it is not ([Keys](./keys#revoking-a-stolen-key)). Keep keys apart from the data, as you would keep any secret, and the vault key apart from the vault backups. Keep the root key offline; it is needed only to issue grants and revocations. ## Restore 1. **Stop the kernel.** 2. **Put back** the event store, the vault, the data directory, the keys and the `.grant` files. 3. **Delete the snapshot store** if it is a SQLite file (`--snapshots sqlite:…`). Snapshots newer than the restored log would not match it; workers rebuild them. 4. **Check the store:** `kernable verify ...` must say `OK`. 5. **Start the kernel** of the build that made the backup. A backup from an earlier development build does not carry over: before the first release, formats change in place and there are no migrations. 6. **Rebuild the read models that have a store** ([Rebuild a read model](./rebuild-read-model)): their stores may be ahead of the restored log. Read models in memory read every event again on start anyway. Everything written after the backup was taken is lost: commands, events, erasures, files. ## Continuous backup The steps above copy the stores now and then: what happened since the last copy is lost. A **backup follower** writes every record to object storage seconds after it is written, and `kernable restore` builds the stores again, up to the newest record or a point in time before something went wrong, such as a faulty deploy at 13:40. 1. **Start the kernel with `--backup on`.** It keeps its record log for the follower, as for a standby. 2. **Make a key for the follower and a backup key:** ```sh kernable keys new backup && kernable grant root.key backup.pub 'backup=*' kernable backup key backup-encryption.key ``` The grant lets the key read every record and file; vault values reach it sealed with the kernel's vault key. The backup key encrypts everything the follower writes (AES-256-GCM): a bucket that leaks reveals nothing. Keep the backup key apart from the bucket, as you keep the root key; without it, no backup can be read. 3. **Run the follower**, next to the kernel or on another machine: ```sh AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… KERNABLE_S3_ENDPOINT=https://fsn1.your-objectstorage.com \ kernable backup --follow https://10.0.0.1:7233,https://10.0.0.2:7233 --key backup.key \ --to s3://acme-backups/shop --backup-key backup-encryption.key ``` `--follow` names both kernels of a pair: the follower turns to whichever is the primary. `--to` is `s3:///` for AWS S3 and S3-compatible storage (Hetzner, MinIO with `KERNABLE_S3_PATH_STYLE=1`), or `file:`. `KERNABLE_S3_ENDPOINT` may carry a path (`https://storage.example.com/s3`); a bucket name with dots needs `KERNABLE_S3_PATH_STYLE=1` over HTTPS, as its name does not match the endpoint's certificate otherwise. The connection uses `KERNABLE_TLS_*`, as the SDKs do. What it writes, in **generations**: a base (a full image of the kernel, or nothing if the record log reaches back to the start), then a segment every `--segment-seconds` (default 5), 16 MiB or 4,096 records. The follower acknowledges records to the kernel only once their segment is in the target, so the kernel keeps them until then. A new generation starts every `--base-hours` (default 24), and the newest `--keep` (default 7) are kept. So a lost primary loses at most the last few seconds that were not uploaded yet, unless a [standby](./standby) holds them. The follower never holds up the kernel: it is not a standby, and a write never waits for it. Stopped and started again, it continues where its last segment ends; if the kernel no longer keeps those records, or its log differs from the backup's, a new generation begins. `kernable backup list s3://acme-backups/shop --backup-key backup-encryption.key` shows the generations, each base's record and time, and how far its segments reach. ### Restore from a continuous backup ```sh kernable restore s3://acme-backups/shop --backup-key backup-encryption.key \ --at 2026-11-04T13:39:00Z --store sqlite:kernable.db --vault-store sqlite:vault.db --vault-key vault.key ``` It fills new, empty stores from the newest generation that begins before `--at` (or `--at-record `; without either, everything), and stops before the first batch the follower took after that time. If the backup ends earlier, or holds nothing between the end of that generation and the base of the next, it says so in a `note:` line and restores up to there. Then it checks the store like `kernable verify`. `--vault-key` must be the source's vault key. Then start the kernel on the restored stores and rebuild the read models that have a store, as in steps 5 and 6 of [Restore](#restore); the note on erasure below applies too. ### Check a backup A backup is only as good as its last restore. `kernable backup check` reads the newest generation all through without writing a store: every object opens with the backup key and decodes, the base holds the records its marker names, and the segments' records chain from the base's hash to the last one without a gap. It prints the record a restore would reach and how old it is, and exits 1 if the backup does not check out: ```sh kernable backup check s3://acme-backups/shop --backup-key backup-encryption.key ``` The follower runs it by itself with `--check-hours ` (`check-hours` in a node's `[backup]`) and logs `the backup checks out` or, as an error, why not: alert on that line. Restore into scratch stores now and then as well (`bench/backup-drill.sh` does it end to end): a check proves the bytes, a restore also the time it takes. ### What it loses, and how long the way back takes Measured with `bench/backup-drill.sh` on a 2-core cloud server (Hetzner, 2026-11-07), the backup in object storage in the same region: a kernel on a store of 200,000 commands takes 200 commands a second, and at a random moment the kernel, the follower and the clients are killed together. | | Segments every 5 s | Segments every 1 s | |---|---|---| | Commands stored but not in the backup | 126 (0.6 s); 759 (3.8 s) in another run | 127 (0.6 s) | | Restore of about 206,000 events, 250 MB, verify included | 63 s | 68 s | | From the restore's start until a kernel on it answers | 64 s | 69 s | A store five times as large (1,006,043 events, 1.17 GB) lost 559 commands (2.8 s) and was back after 319 s: the way back grows with the store, about linearly. So the point in time a restore reaches is at most `--segment-seconds` and an upload behind the loss. The restore's time grows with the store: most of it goes to applying the records and checking every signature (on every core). A [standby](./standby) loses nothing a client was told and takes over in seconds; the backup is for the machine and the standby lost together, for mistakes, and for keeping history elsewhere. ## Backups and erasure A backup keeps personal data that was erased after it was taken. Keep vault backups only as long as your retention rules allow. The kernel does not repeat erasures by itself. If you restore a vault that is older than the event store, or older than an erasure that happened since, delete the erased values again. The log names every erased subject in the `erases` column of its erasing events. With the kernel stopped: ```sh sqlite3 vault.db "ATTACH 'kernable.db' AS log; DELETE FROM vault WHERE secret = 0 AND EXISTS ( SELECT 1 FROM log.events e WHERE e.erases = vault.subject AND e.tenant_id = vault.tenant AND e.created_at >= vault.created_at);" ``` This deletes the values of each erased subject that were written up to its erasure, and nothing written later. Then delete the snapshot store, as in step 3 above. Files are never erased with a person; delete them yourself ([Handle an erasure request](./erasure)). --- # Move a kernel with export and import `kernable export` writes everything a kernel holds into one file, and `kernable import` fills new stores from it. Use it to: - **leave a hosted kernel** and run it yourself, with the same commands, events, vault values and files; - **move to another machine**, or **change the store backend** (SQLite to the `file:` log, or back); - **hand a copy to someone** who checks it with `kernable verify`, without access to your machines. For regular backups, see [Back up and restore](./backup); an export is a full copy each time. ## What an export holds | Part | In the export | Notes | |---|---|---| | the event store | every command, event, rejection, checkpoint, revocation, grant, membership, registration of listeners and workers, with their positions | as one consistent view | | the vault | every value the store refers to | **still encrypted with the source's vault key**: the file holds no personal data in clear | | files | every file of the data store | as stored | | snapshots, the cache | no | workers rebuild snapshots; cache values expire | | keys | no | the export names the kernel keys that signed its checkpoints, by key ID | A file never holds a value that was erased before the export. It does hold values erased **after** it: delete an export once it is imported, as you would an old backup ([Backups and erasure](./backup#backups-and-erasure)). ## Export a running kernel The kernel streams the export while it serves; nothing stops. The key needs a grant for `export`: ```sh kernable keys new ops && kernable grant root.key ops.pub 'export=*' kernable export 127.0.0.1:7233 ops.key kernable.kx ``` ```text exported 48112 records, 913 vault values and 27 files to kernable.kx signed by the kernel keys 3f9c… ``` The kernel signs a checkpoint first, so every event in the file is anchored. Run it over mutual TLS (set `KERNABLE_TLS_CA`, `KERNABLE_TLS_CERT`, `KERNABLE_TLS_KEY`): the stream carries everything. A standby does not export; ask the primary. The file is created readable by its owner only, never overwrites one, and is deleted if the export does not finish. ## Export stopped stores With the kernel stopped, read the stores directly. The flags are those of `kernable start`, with its defaults: ```sh kernable export kernable.kx --store sqlite:kernable.db --data-store file:data \ --vault-store sqlite:vault.db --vault-key vault.key ``` Without `--vault-key` the vault is left out, which fails if the store refers to any vault value. ## Import Import into **new, empty** stores, with the source's vault key: ```sh kernable import kernable.kx --store file:kernable.log --data-store file:data \ --vault-store sqlite:vault.db --vault-key vault.key --kernel-pub old-kernel.pub ``` ```text imported 48112 records, 913 vault values and 27 files 41870 events, 31402 commands, 412 checkpoints, 0 events after the last checkpoint OK ``` The import reads the whole file once before it writes anything: a copy cut short or changed (the file ends with a SHA-256 of everything before it), a file of another format, or a vault key that does not open its values is refused. Then it checks the new store like `kernable verify`, with the same output and exit codes: `OK, 1 suspect` (exit 2) is what the source would show too. Name the source's kernel keys with `--kernel-pub`. Without them, the store is checked with the keys the export names, which proves the file is consistent but not who signed it: compare the printed key IDs with the source's `.pub` files. ## Start the new kernel ```sh kernable start --root root.pub --store file:kernable.log --vault-key vault.key ... ``` - **The root key:** grants are imported with the store and stay valid only under the root key that signed them. Start with the same `--root`, or grant every key again with a new one ([Keys](./keys)). - **The kernel key:** use the source's, or a new one ([Keys](./keys#the-kernel-s-key)). Keep the source's `.pub` files: `kernable verify` needs every key that ever signed a checkpoint. - **The vault key:** the import kept the values encrypted with the source's key, so the new kernel uses it. To change it afterwards, run `kernable vault rekey` ([CLI](../reference/cli)). - **Listeners** keep their positions. Read models with their own store carry on where they were if they come along; new ones read every event from the start. --- # Run a standby A **standby** is a second kernel that holds every write the primary told anyone about. If the primary's machine is lost, you promote the standby and it serves on, without losing a committed command, an acknowledged listener position or a confirmed file. What you get: - **Nothing told is lost.** The primary answers a client, delivers an event, or confirms a file only once the standby holds it on its disk. - **Automatic failover with a witness**, a small third process: the standby takes the primary's place within seconds, and the old primary comes back as its standby by itself. Without a witness, `kernable promote` does it by hand. - **Clients find the primary themselves.** Give every process both kernels' addresses, comma-separated (`KERNABLE_URL=https://10.0.0.1:7233,https://10.0.0.2:7233`): a request or stream that meets the standby (`not_primary`) or a kernel that is gone goes to the other one. Commands are idempotent by ID, so a resubmit after a failover gets the original outcome. Two ways to run it: [a pair with a witness](#a-pair-with-a-witness) (recommended), or [a fixed primary and standby](#a-fixed-primary-and-standby) that you promote by hand. ## How it works The primary sends the standby every record of its store, in order and chained by a hash, with the vault values and files they refer to. The standby checks every record before it writes it, and acknowledges what is on its disk. With `--replication sync` the primary waits for that acknowledgement before it tells anyone of a write. Design: `design/replication.md`. The standby serves no requests: every call gets `UNAVAILABLE` with `not_primary`, and its `/ready` answers 503. The SDKs retry `UNAVAILABLE` by themselves. ## A pair with a witness Three machines: two kernels and the witness. The witness keeps only who is the primary and whether its standby is in sync, in one small file; it needs no store and little of anything. ```sh kernable keys new node-a && kernable grant root.key node-a.pub 'replicate=*' kernable keys new node-b && kernable grant root.key node-b.pub 'replicate=*' # machine W kernable witness --root root.pub --listen 10.0.0.9:7300 --state witness.state \ --tls-cert … --tls-key … --tls-client-ca … # machines A and B, the same flags but their own keys, stores and addresses kernable start --root root.pub --witness 10.0.0.9:7300 \ --node-key node-a.key --advertise 10.0.0.1:7233 \ --primary-kernel-key kernel-a.pub --primary-kernel-key kernel-b.pub \ --kernel-key kernel-a.key --vault-key vault-a.key \ --listen 10.0.0.1:7233 --tls-cert … --tls-key … --tls-client-ca … ``` - Start the kernel that holds the data first: on a new witness, the first kernel to ask becomes the primary of term 1. The other one follows it, taking an image if it is empty. - `--advertise` is where the other kernel reaches this one; `--primary-kernel-key` names both kernels' kernel keys, since either may be the primary. - The kernels reach the witness and each other with the `KERNABLE_TLS_*` variables, as the SDKs do. **What happens:** | When | What the pair does | |---|---| | Both run | The primary renews its lease with the witness every second (`--lease-ms`, 3,000 by default, at the witness). Writes wait for the standby. | | The primary's machine is lost | The standby loses its stream. Once the primary's lease has expired at the witness (3 s, plus 1 s for clocks), the standby claims its place, becomes the primary of the next term, and serves. | | The standby is away or behind for `--standby-timeout-ms` (5 s) | The primary tells the witness, then commits alone. The witness no longer lets the standby take over: it lacks those writes. Back, the standby catches up; the primary waits for it again, then tells the witness it is in sync. | | The old primary comes back | The witness names the new primary; the old one follows it. Writes it never told anyone of make its log differ; it moves its stores aside and takes an image. | | The primary loses the witness | In sync, it goes on: the standby's acknowledgements fence it. Alone, it stops telling writes once its lease has run out (a margin before the witness would let anyone take over), until it reaches the witness again. | | The witness is down | No failover, and the primary cannot go alone; writes with the standby in sync go on. | | A primary learns it was replaced | It stops with an error. Run kernels under a supervisor (systemd, Kubernetes): started again, it follows the new primary. | `kernable promote` on the standby of a pair asks the witness, too: it is refused while the primary holds its lease. Keep the witness on a third machine, so a lost machine never takes the witness and a kernel down together. Its state file decides who is the primary: back it up, and never start a pair against a new, empty witness while the old primary may still run. ### Update a pair without downtime With `kernable switchover`, the primary hands its role over to the standby at once, without waiting for its lease to run out: 1. **Update the standby**: stop it, install the new binary, start it again; it catches up and is in sync again (`kernable_standby_in_sync` on the primary). 2. **Switch over**, with a key granted `manage` for `*`: ```sh kernable switchover https://10.0.0.1:7233 admin.key ``` The primary refuses new requests (`UNAVAILABLE`: clients with both addresses turn to the other kernel), lets running ones finish, waits until the standby holds every record, and tells the witness. The standby serves at once, without losing a told write; the old primary stops. 3. **Update the old primary** and start it: it follows the new primary as its standby. If the standby is not in sync, or does not hold everything within 10 s, the switchover is refused and the primary serves on. Under [`kernable node`](./node), the stopped kernel is started again by itself. A **planned stop** does the same: SIGTERM to the primary of a pair (`systemctl stop`, a node stopping) hands its role to the standby in sync first, then the primary drains and stops. The standby serves at once instead of after the lease. Without a standby in sync, the primary stops as before. A primary of a pair whose lease at the witness ran out while no standby in sync holds its writes (cut off by the network, say) tells nothing more: a write waiting for the standby, and every new request, is answered `UNAVAILABLE` within about a lease, so clients with both addresses turn to the other kernel and submit again there (safe: commands are idempotent by ID). While it is fenced like this it writes nothing in the background either (no checkpoint, listener position or worker), so all its log can hold that the new primary lacks are the commits that were under way. Its standby notices the silence within two thirds of a lease (pings on its connection) and takes over. A standby stopped with SIGTERM tells its primary that it leaves, and the primary of a pair commits alone at once instead of after `--standby-timeout-ms`. SIGUSR1 stops a kernel without handing over; [`kernable node`](./node) sends it when a new kernel binary on the same machine takes the stores over. ## A fixed primary and standby Without a witness, the roles stay as you start them: you promote the standby by hand when the primary is lost, or hand the role over with a [switchover](#switch-over) while both run. ## 1. Keys The standby needs three keys of its own: ```sh kernable keys new standby # proves the standby to the primary kernable grant root.key standby.pub 'replicate=*' kernable keys new standby-kernel # signs the checkpoints it writes once promoted kernable keys new standby-vault # encrypts its vault ``` `replicate` lets a key receive everything: every event, command and file, and every vault value in plaintext. Keep `standby.key` as carefully as the root key, and run replication over mutual TLS only (see [Mutual TLS](./tls)). Revoke it, and the standby's stream ends at once; in a pair, the primary commits alone, so the revoked key can never take its place. The standby encrypts its vault with its own key; no secret is copied between the machines. ## 2. Start the primary with `--replication sync` ```sh kernable start --root root.pub --replication sync \ --listen 10.0.0.1:7233 --tls-cert … --tls-key … --tls-client-ca … \ --vault-key vault.key ``` Until a standby has caught up once, the primary serves alone, and the standby takes its image and catches up meanwhile without holding up a write. Once it holds every write the primary told, the primary waits for it from the next write on, notes so beside its store (`.standby-caught-up`, so that started again it waits at once), and tells the standby it is in sync. From then on, writes wait without a standby in sync and `/ready` answers `waiting for a standby in sync`. ## 3. Start the standby On the second machine, with its own stores: ```sh KERNABLE_TLS_CA=ca.crt KERNABLE_TLS_CERT=standby.crt KERNABLE_TLS_KEY=standby.tls.key \ kernable start --root root.pub \ --follow 10.0.0.1:7233 --follow-key standby.key --primary-kernel-key kernel.pub \ --kernel-key standby-kernel.key --vault-key standby-vault.key \ --listen 10.0.0.2:7233 --tls-cert … --tls-key … --tls-client-ca … ``` - `--follow` is the primary's address; the connection uses the `KERNABLE_TLS_*` variables, as the SDKs do. - `--primary-kernel-key` is the primary's kernel public key: the standby checks the primary's checkpoints with it. Give every key the primary had, if it changed its key. - Start it with the flags it should have as the primary later, such as `--replication sync` if it should wait for a standby of its own once promoted. An **empty** standby takes a full image of the primary first, then follows. The first standby of a primary does so while the primary serves alone. A standby that replaces one that caught up before makes the primary's writes wait while it takes the image: with a large store, add it in a quiet hour. To let it catch up without the wait instead, stop the primary, delete `.standby-caught-up`, and start it again; from then on, the old standby must not be promoted. A standby that was away only briefly catches up from where it stopped: the file log keeps every record, and SQLite keeps 100,000 records beyond what the standby acknowledged. ## 4. Watch it | Metric | On | Meaning | |---|---|---| | `kernable_standby_in_sync` | primary | 1 while writes are told: a standby is connected and has caught up, or the primary serves alone until its first standby has | | `kernable_standby_catching_up` | primary | 1 while no standby has caught up yet: the primary serves alone | | `kernable_standby_acknowledged`, `kernable_record_head` | primary | the newest record the standby holds, and the newest written; writes in between wait | | `kernable_standby_lag` | standby | records behind the primary's head as last seen | Alert when `kernable_standby_in_sync` is 0: the primary's writes wait. The primary also logs `a write waits for the standby` every 10 s while one waits. ## When the standby is away The primary's writes wait. Commands stay queued and submits time out (`--submit-timeout-ms`), but nothing is lost: once the standby is back, it catches up and the queued commands run. If the standby is lost for good and the primary must serve alone, restart the primary without `--replication sync`. From then on, that standby must not be promoted: it lacks what the primary writes alone. When you start it again (or a new one), it notices and takes an image. ## Promote the standby Only when the primary is gone, or must be replaced: ```sh kernable promote 10.0.0.2:7233 ops.key # ops.key: granted manage for * # promoted 10.0.0.2:7233: it serves as the primary from record 48213 ``` The standby stops following, opens its stores as a kernel does after a restart, and serves: queued commands run, listeners resume after their stored positions, and a client that resubmits a command gets the original outcome. Snapshots start empty: the first command per aggregate reads its whole history. Processes given both addresses turn to it by themselves. Others: change the address they use (DNS, a load balancer that checks `/ready`), and restart them if they do not pick up the change. The standby refuses promotion if it never followed a primary, if it is in the middle of an image, or if the primary never told it it is in sync (`.in-sync` beside its store is missing): the primary may then have told writes it lacks. The note stays when the standby starts again, so a standby whose primary was lost can still be promoted; it goes when it reaches a primary again, which tells it anew once it has caught up. ## Switch over To move the primary role to the standby on purpose (an update, or a move to another machine), while both run: ```sh kernable switchover 10.0.0.1:7233 ops.key # ops.key: granted manage for * # switched over: is the primary now, from record 48213; 10.0.0.1:7233 stops … ``` The primary refuses new requests (the SDKs retry them on the standby), lets running ones finish, waits until the standby holds every record (at most 10 s), and tells it to take over on its stream. From that record on it writes nothing more, not a checkpoint, a listener's position or a worker's registration (its record log is sealed; a worker registering meanwhile gets `UNAVAILABLE` and tries the standby), so its own log stays the beginning of the new primary's. The standby answers and serves at once; the old primary stops. It notes beside its store that it handed over (`.handed-over`): started again as a primary, it refuses, so it never serves next to the new one by mistake. Start it with `--follow` and the new primary's address to make it the new standby. If the old primary stops before the standby heard it (it crashed in between), the standby waits for its primary: promote it by hand, it holds every record the primary told. The switchover is refused while no standby is in sync, and if the standby does not catch up within 10 s the primary serves on. ## Bring the old primary back Never start the old primary as a primary again; after a switchover it refuses to. Start it as a standby of the new one, with `--follow` and the new primary's kernel key. After a switchover or a planned stop it follows on from its own log. Cut off by the network, it may hold writes it never told anyone of (the commits under way when it lost its standby); then its log differs from the new primary's, and it takes exactly those writes back and follows on (`the standby took back its records the primary does not hold`). Only if one of them cannot be taken back (an erasure, a data file's change, a completed command whose vault values are gone) it moves its stores aside (`.diverged-`) and takes an image. Only the newest copy stays: a kernel that starts over again removes the one before, so they take one store's size at most. The files it moves aside can hold personal data: delete them once you know nothing in them is needed. A primary that still ran when you promoted its standby tells nobody of anything it writes from then on: no standby acknowledges it any more. ## What it costs Every write waits for one more round trip: to the standby, its own flush, and back. Measured on one Apple M2 Max with both kernels on the same machine and disk, a Rust worker, 1 KB payloads, `wait: committed` (`bench/replication.sh`, results in `bench/`): | Store, durability | Alone, 1 client | With a standby, 1 client | Alone, 64 clients | With a standby, 64 clients | |---|---|---|---|---| | SQLite, `full` | 167/s | 84/s | 3,094/s | 659/s | | SQLite, `full`, standby `normal` | 161/s | 153/s | 3,042/s | 2,018/s | | SQLite, `normal` | 2,823/s | 1,702/s | 7,154/s | 4,090/s | | file log, `normal` | 3,711/s | 2,535/s | 12,833/s | 7,253/s | With `full` durability on one disk, the two kernels' flushes queue behind each other; the row with a `normal` standby shows the cost without that, which is closer to two machines with their own disks. Measure on your own machines before you size them. ## A standby is not a backup A standby copies every write, mistakes included, within milliseconds. Keep your [backups](./backup). --- # Run a listener on several machines Read this when the same read models and reactors run on more than one machine, for example on both machines of a [standby pair](./standby): each machine keeps its own copy of every read model, so a lost machine takes nothing with it. Several processes then run one listener under one name, each as an **instance**. The kernel treats them as one listener: - **It is through an event once one instance has handled it.** A settling submit, a listener of a later order and `/ready` wait for the fastest instance, not for all of them. An instance that is gone, or rebuilding from the first event, holds nothing back. - **Read models are replicated.** Every instance gets every event, from its own position, and keeps its own state. A query goes to any instance that has reached the query's `min_position`, so it sees what the submit before it wrote, whichever instance answers. A middleware check goes to any instance that has caught up. - **Reactors are single.** One instance at a time gets events; the others wait. When the active one goes away, the first waiting one takes over after the last event the group acknowledged, so an email is sent once, not once per machine. Delivery stays at least once: the events the old instance had not acknowledged come again. ## Name the instance Set `KERNABLE_INSTANCE` for each process, for example to the machine's name: ```sh # machine A KERNABLE_INSTANCE=node-a KERNABLE_URL=https://10.0.0.1:7233,https://10.0.0.2:7233 node app.ts # machine B KERNABLE_INSTANCE=node-b KERNABLE_URL=https://10.0.0.1:7233,https://10.0.0.2:7233 node app.ts ``` An instance ID has 1 to 128 letters, digits, `-`, `_` and `.`, and is unique within its listener. The code does not change: every read model and reactor of the process joins its group. Without `KERNABLE_INSTANCE`, a listener has one connection, and a second process with the same listener is refused (`ALREADY_EXISTS`). To set it per listener instead, or to change the kind, use the listener's options: `instance` and `single` in TypeScript (`ListenerOptions`) and Python (`read_model`, `reactor`), `.instance(…)` and `.single(…)` in Rust. A read model is replicated (`single: false`) unless you say otherwise; a reactor is single. A live listener is never single: each process gets every event, for its own connected browsers. ## What to keep in mind - **Every instance of a listener agrees on its kind.** A replicated instance next to a single one is refused (`FAILED_PRECONDITION`, `disagree on single`): change both, or neither, in one deploy. - **A replicated listener cannot park events.** Each instance has its own copy, and a parked event would be missing from one copy only; read models do not park by default. Make a listener single if it parks. - **A read model with a store keeps its position on its machine.** Back on, it continues after its own stored position and gets what it missed. One in memory rebuilds from the first event, as on every start. - **Both machines run every reactor's code, one at a time.** A reactor whose side effects must not happen from a second place at all, such as a scheduled job that ignores events, belongs in one process only. - **The newest registration counts.** Instances normally run the same code. During a rolling deploy they may differ in order, event types or middleware for a while: the kernel takes the registration of the instance that connected last, and logs a warning, in case the difference lasts. - **At most 64 instances** of one listener are connected at once. The browser UI shows a listener's connected instances, each with its position; `kernable listeners` reads the store, which keeps only the group's position. --- # Run a node from one file `kernable node` runs one instance of your deployment on a machine from a single `project.toml`: the kernel, the [backup follower](./backup#continuous-backup) if you want it, and your own services (workers, read models, reactors, apps) as [Podman](https://podman.io) containers it builds on the machine. On Linux the kernel and the backup follower run in Podman too, all in one pod per instance, so [limits](#limits) hold for every part and for the instance as a whole. It looks after all of them as child processes. It is the same in every place kernable runs, on your own machines or someone else's. - **A child that exits is started again**, after 1 s, backing off up to 30 s while it keeps failing at once. A kernel crash never takes the node down. A child being stopped for a restart stops in the background: the node looks after the others meanwhile, and starts it again once its old process ended. - **A change to the file is applied by itself.** The node reads it every two seconds (or at once on SIGHUP); a child whose command changed is stopped and started again with the new one, and the others run on. - **An update is a new binary.** Point `binary` at it, or replace the file it names: the node stops the child and starts it on the new binary. The kernel is not stopped first: the new kernel starts beside it and waits for the stores, the old one gets SIGUSR1, finishes its running requests and lets them go, and writes pause for a moment only. A standby takes its stores over the same way, so both kernels of a pair update one after the other and keep their roles. - **A new build of a service on SIGHUP.** Change your code, send the node SIGHUP: it builds every service again in the background, and restarts those whose image changed. The other children are looked after meanwhile. - **One log for everything.** Every line of every child comes out of the node, on stdout or stderr as the child wrote it, prefixed with its name (`worker | …`, `kernel | …`). Under systemd, journald keeps, rotates and searches them: `journalctl -u kernable-node | grep 'worker |'`. - **Metrics** with `metrics-listen` in `[node]`: see [below](#metrics). - **SIGHUP also goes to the kernel**, which reads its keys again (and runs `--key-command` again). - **SIGTERM or SIGINT stops everything**: your services and the backup follower first, then the kernel, each with SIGTERM, and SIGKILL after `stop-seconds` (default 30) if it has not stopped by then. The kernel drains as it does on its own ([Deploy and upgrade](./deploy)). ## The file ```toml [kernel] binary = "/opt/kernable/1.0.1/kernable" # optional: default the binary that runs the node stop-seconds = 30 # optional flags = { root = "root.pub", store = "sqlite:kernable.db", vault-key = "vault.key" } env = { KERNABLE_LOG = "info" } # optional cpus = "2" # optional (Linux), as memory = "2g" image = "docker.io/library/debian:trixie-slim" # optional (Linux): the base image, this one by default [backup] # optional to = "s3://acme-backups/shop" key = "backup.key" backup-key = "backup-encryption.key" follow = "10.0.0.1:7233,10.0.0.2:7233" # optional: default the kernel's listen segment-seconds = 5 # optional, as base-hours and keep check-hours = 24 # optional: reads the backup through that often env = { AWS_ACCESS_KEY_ID = "…", AWS_SECRET_ACCESS_KEY = "…" } [pair] # optional: one node of a pair witness = "10.0.0.9:7300" node-key = "node-a.key" advertise = "10.0.0.1:7233" # optional: default the kernel's listen peer = "10.0.0.2:7233" # the other node's kernel kernel-keys = ["kernel-a.pub", "kernel-b.pub"] [node] # optional instance = "node-a" # KERNABLE_INSTANCE for the services (default the host name) project = "shop" # names the containers (default the file's directory) metrics-listen = "127.0.0.1:7240" # optional: GET /metrics cpus = "4" # optional (Linux): the whole instance, as memory = "8g" network = "host" # optional (Linux): see Network; default own [[service]] # one per service, as many as you need name = "worker" build = "./worker" # a directory with a Containerfile or Dockerfile command = ["node", "app.ts"] # optional: the image's own by default key = "keys/worker.key" # optional: with keys/worker.grant next to it env = { SHOP_MAIL = "smtp://mail:25" } # optional secrets = ["stripe"] # optional: see [[secret]] cpus = "1.5" # optional, as memory = "512m" stop-seconds = 10 # optional writable = true # optional: see Hardening expose = 3000 # optional (Linux): see Network [[secret]] # one per secret name = "stripe" file = "secrets/stripe.key" # or: command = ["vault", "kv", "get", "-field=key", "shop/stripe"] ``` `flags` are the flags of `kernable start` ([Configuration](../reference/configuration)) without their dashes; `dev = true` stands for `--dev`. With a `[backup]` section the kernel gets `--backup on`; it takes `cpus` and `memory` too. Paths are relative to the file's directory, where the children run; on Linux every path the kernel and the backup follower use lies inside it, since that directory is all their container sees. Unknown keys are refused, so a typo does not go unnoticed: at start the node exits with 2, later it logs the error and the children run on as they are. ```sh kernable node project.toml ``` Run it under your service manager (systemd, launchd) so the node itself starts with the machine. The children run in process groups of their own, so a Ctrl-C reaches only the node, which stops them in order; under systemd, set `KillMode=mixed`, so SIGTERM goes to the node alone, and a `TimeoutStopSec` longer than the children's `stop-seconds` together. On Linux, a node that is killed takes its children along: a small watchdog process removes the instance's pod once the node is gone, and a node that starts again replaces whatever pod an earlier one left. ## Your services Each `[[service]]` is built with `podman build` on the machine, at start, when its section changes, and on SIGHUP, then run with `podman run --rm` as `kernable--`, after the kernel and stopped before it. The node builds one image at a time, and stops a build that takes over 20 minutes; a build that fails or is stopped is tried again after a pause, and a service that already runs keeps its image meanwhile. It finds the kernel by itself: the node sets `KERNABLE_URL` to the kernel's socket (on Linux, see [Network](#network)) or its `listen`, and `KERNABLE_INSTANCE` to the node's instance, which the SDKs read. With `key`, the key file and the `.grant` file next to it are mounted read-only and named in `KERNABLE_KEY` and `KERNABLE_GRANT`; create them with `kernable keys new` and `kernable grant`, so the root key never comes near the machine. A service that exits is started again like the kernel. Install Podman on the machine first (rootless is fine). Services run on Linux; on a Mac, Podman's containers live in a virtual machine and do not share its network. ## Limits On Linux every part of the instance runs in Podman, in one pod named `kernable-`: - **The kernel and the backup follower** run in a small base image with the node's kernable binary mounted read-only, so an update is still a new binary. The project directory is mounted at its own path. Name your own base image in `[kernel] image`, for one with your KMS's client for `--key-command`, say. Podman pulls it on the first start. - **Each part** takes `cpus` and `memory`: `[kernel]`, `[backup]`, every `[[service]]`. - **The instance as a whole** takes them in `[node]`: the kernel and every service together never use more, so instances side by side on one machine cannot starve each other. A change to these restarts every part in a new pod. - **Variables** in `env` reach the containers from Podman's own environment, so their values never show in a process list. On macOS and other systems the kernel and the backup follower run as plain processes, without limits; asking for them there is an error. ## Network On Linux the instance's pod has a network of its own, so its services see neither the machine's other services nor other instances, and nothing in it can be reached from outside: - **The kernel listens on a socket**, `run/kernel.sock` in the project directory; the node chooses it, so `flags` takes no `listen`. Each service finds it, read-only, at `/run/kernable/kernel/kernel.sock`, named in `KERNABLE_URL`. - **A service that serves HTTP** (a web frontend, an API, webhooks) takes `expose = `, the port it listens on in the pod. The node passes `run/.sock` on to it, so the service listens on plain TCP as usual and is reached through its socket. - **From outside**, a reverse proxy on the machine ends TLS and passes requests on to these sockets: gRPC to `run/kernel.sock` (HTTP/2 without TLS: `grpc_pass unix:…` in nginx), HTTP to `run/.sock`. The sockets let their owner and group connect; grant your proxy's user access with an ACL (`setfacl -m u::x` on the directory and `u::rx,d:u::rw` on `run/`). One of a [pair](#two-machines) names in `[pair] advertise` where the other node reaches its kernel through that proxy (`host:443`). - **Your services reach the internet** as usual (rootless, through Podman's pasta). The machine's loopback is out of their reach. Run each instance as a user of its own, and keep its user from the machine's other addresses and the provider's metadata service with your firewall. `network = "host"` puts the whole pod on the machine's network, as on a machine of its own, where nothing else runs beside the instance: the kernel listens where `listen` says (plain TCP on a loopback address only, mutual TLS elsewhere), and the services find it there. A change of `network` starts every part again in a new pod. ## Hardening Every container the node starts, the kernel's included, gets the same protection, so that code which takes over a process gets as little as possible: - **No capabilities**, and **no way to gain privileges**: a setuid binary in the image does not work. - **At most 4096 processes**, so a fork bomb stays inside its container. - **A read-only root filesystem**, with a fresh, empty `/tmp`, `/var/tmp` and `/run` in memory. What has to last belongs in the kernel. - **On Linux, Podman's system call filter, tightened**: no new user namespaces, whose "root" would hold capabilities again and reach the kernel paths most container escapes go through, and no kernel keyrings, which nothing separates between containers. The node writes the profile from Podman's own (`/usr/share/containers/seccomp.json`, or yours in `/etc/containers/`) when it makes the pod, so Podman's updates reach it; without one it does not start. Most images run as they are. One that writes elsewhere (some frameworks keep a cache next to their code) takes `writable = true` in its `[[service]]`; the node logs it at each start. Images that start as root to switch to a user of their own, such as the official `nginx`, do not start without capabilities: use an image that runs as an ordinary user from the start (`nginxinc/nginx-unprivileged`, say). Software that builds a sandbox of its own from user namespaces runs without it, as is usual in containers: Chromium (Puppeteer, Playwright) with `--no-sandbox`; containers inside a service do not run. ## Secrets A `[[secret]]` names a value your services need (a payment provider's key, a mail password) without writing it into the file: it comes from a file, or from the output of a command such as your KMS's client. A service that lists it in `secrets` finds it, read-only, at `/run/secrets/`. - **A file** is read again when it changes; **a command** runs at start and on SIGHUP, as `--key-command` does. - **A new value** restarts the services that read it, and only those. - **Where the values live:** in a directory only the node's user may enter, in memory where the system has one (`$XDG_RUNTIME_DIR`, or `/run` as root). - **A secret that cannot be read** stops the node at start; later, the services keep the value they have, and the log names the secret. ## Metrics With `metrics-listen`, `GET /metrics` answers in Prometheus' text format, a line per child: | Metric | Type | What | |---|---|---| | `kernable_node_child_up{child}` | gauge | 1 while the child runs | | `kernable_node_child_restarts_total{child}` | counter | times the node started it again | | `kernable_node_child_uptime_seconds{child}` | gauge | seconds since it last started, while it runs | | `kernable_node_container_cpu_seconds_total{child}` | counter | CPU time of the child's container: a service, and on Linux the kernel and the backup follower (from its cgroup, every 10 s) | | `kernable_node_container_memory_bytes{child}` | gauge | memory of the child's container now | The kernel keeps its own metrics on its `--metrics-listen` ([Monitor kernable](./monitor)). The address is read when the node starts; a change applies after a restart of the node. ## Two machines Each machine of a [standby pair](./standby) runs its own node, with its own `project.toml` and a `[pair]` section: the witness, this node's key, its own address and the other node's (`peer`), and both kernels' kernel keys. From it the node sets the kernel's pair flags (don't repeat them in `flags`), lets the backup follower turn to whichever kernel is the primary, and gives your services both addresses in `KERNABLE_URL`. Your read models and reactors run on both machines as [instances](./instances). In the pod's own network, `advertise` is required: the address of this machine's reverse proxy that leads to this kernel's socket (`a.example.com:443`), and `peer` the other machine's. The kernel reaches the other one and the witness over TLS with the files its `env` names (`KERNABLE_TLS_CA`, `KERNABLE_TLS_CERT`, `KERNABLE_TLS_KEY`, inside the project directory), and so do your services that turn to the other node; the backup follower stays on this machine's socket. --- # Run kernable on your own servers This page takes a bare Linux machine (Debian 12 or Ubuntu 24.04 and newer, x86_64 or arm64) to a kernel that starts with the machine, keeps its data on a disk you chose, and runs your services next to it. It is the same `kernable node` that runs kernable anywhere else; what it adds is the machine around it: a user, a directory, a systemd unit, a firewall. ## 1. Install ```sh curl -fsSL https://kernable.com/install.sh | sudo KERNABLE_INSTALL=/usr/local/bin sh kernable --version ``` The script picks the archive for the machine, checks it against the release's `SHA256SUMS`, and installs one file. For your services the node needs Podman (rootless) and, on Linux, runs the kernel in it too: ```sh sudo apt-get install -y podman uidmap passt ``` ## 2. A user and a directory The node runs as a user of its own, never as root: ```sh sudo useradd --system --create-home --home-dir /srv/kernable --shell /usr/sbin/nologin kernable sudo usermod --add-subuids 200000-265535 --add-subgids 200000-265535 kernable # for rootless Podman sudo loginctl enable-linger kernable # its units run without a login sudo systemctl start user@$(id -u kernable) # its own systemd, now sudo -u kernable mkdir -p /srv/kernable/shop ``` Put `/srv/kernable` on the disk you want the data on. The store is a SQLite file there: back up the directory's disk as you would a database's, and set up [continuous backup](./backup#continuous-backup) besides. ## 3. Keys Make the **root key on your own machine**, not on the server: it signs every grant, and whoever has it decides who may do what ([Trust](../concepts/trust)). Copy only its public half to the server: ```sh kernable keys new root # on your machine: root.key stays there scp root.pub server:/tmp/ && ssh server 'sudo install -o kernable -m 644 /tmp/root.pub /srv/kernable/shop/' ``` On the server, as the user, make the kernel's own keys: the kernel key signs checkpoints, the vault key encrypts personal data ([Passwords and other secrets](./secrets)): ```sh cd /srv/kernable/shop sudo -u kernable kernable keys new kernel sudo -u kernable kernable keys new vault ``` Each service gets a key and a grant from the root key ([Keys per service](./keys)): make the key where the service runs, sign its grant on your machine, copy the grant back. ## 4. The node's file `/srv/kernable/shop/project.toml`, owned by the user ([Run a node from one file](./node) has every setting): ```toml [kernel] flags = { root = "root.pub", kernel-key = "kernel.key", vault-key = "vault.key", store = "sqlite:kernable.db", listen = "unix:/srv/kernable/shop/kernel.sock", metrics-listen = "127.0.0.1:9233" } [node] project = "shop" network = "host" # the machine is the instance's alone: the kernel listens where listen says [[service]] name = "worker" build = "./worker" key = "keys/worker.key" ``` With `network = "host"` the instance is on the machine's network, as on a machine of its own: the kernel listens where `listen` says, its services find it there, and `/ready` answers on the loopback address. Clients on other machines reach it over TCP with mutual TLS only (`listen = "0.0.0.0:7233"` with the `tls-*` flags): plain TCP stays on the loopback address ([Mutual TLS with your own PKI](./tls)). Several instances on one machine each get a pod network of their own instead, the kernel on `run/kernel.sock` and a reverse proxy in front ([Network](./node#network)). ## 5. Start it with the machine A systemd user unit for the user, `/srv/kernable/.config/systemd/user/kernable-node.service`: ```ini [Unit] Description=kernable node shop [Service] WorkingDirectory=/srv/kernable/shop ExecStart=/usr/local/bin/kernable node /srv/kernable/shop/project.toml Restart=always # The node stops its children in order, each within its stop-seconds. TimeoutStopSec=120 [Install] WantedBy=default.target ``` ```sh sudo -u kernable mkdir -p /srv/kernable/.config/systemd/user # write the unit above to that directory, owned by kernable, then: sudo systemctl --user -M kernable@ daemon-reload sudo systemctl --user -M kernable@ enable --now kernable-node sudo journalctl _UID=$(id -u kernable) -f # the node's log: every child, prefixed with its name curl -s 127.0.0.1:9233/ready # ready, or what the kernel waits for ``` `systemctl --user -M kernable@ stop kernable-node` stops it the way a deploy should: the services first, then the kernel, which finishes its running requests ([Deploy and upgrade](./deploy)). ## 6. The firewall Open what clients on other machines use, nothing else: SSH, and the kernel's TLS port if it listens on one. The metrics address stays on the loopback address; scrape it from the machine, or through an SSH tunnel. ## 7. Then - **Updates** are a new binary: install it, point `[kernel] binary` at it or replace the file, and the node hands the kernel's stores over to it without stopping ([Run a node from one file](./node)). - **A second machine** that takes over when this one is lost: a standby, or a pair with a witness on a third machine ([Run a standby](./standby)). - **Backups** to object storage, checked by themselves (`check-hours`), with a restore drill now and then ([Back up and restore](./backup)). - **Watching it**: `/ready` and `/metrics` on the metrics address, alerts worth having, the UI ([Monitor kernable](./monitor)). --- # Share tenants across kernels When one kernel is not enough for all your tenants, run several, each serving a share of them: **partitions**. A tenant lives on exactly one partition, so everything kernable guarantees within a tenant holds as before: one writer per aggregate, unique keys and requirements, listener order, settling, erasure. A signed **partition map** says where each tenant lives, and the SDKs send every request to its tenant's partition by themselves. Every partition is an ordinary kernel, or a [pair with a standby](./standby). One of them, the **control partition**, also holds the map and the trust records (grants, revocations, memberships); the others take them from it every second. ## 1. Keys Every partition takes the trust records, and reserves global unique keys, with a node key granted `replicate` (a pair's node key does both). The control partition needs one too, to ask the others about global keys: ```sh kernable keys new node-p2 && kernable grant root.key node-p2.pub 'replicate=*' ``` Every partition trusts the same root keys (`--root`). ## 2. Start the partitions ```sh # p1: the control partition (no --control) kernable start --root root.pub --partition p1 --listen 10.0.1.1:7233 … # p2: takes the map and the trust records from p1 kernable start --root root.pub --partition p2 --control 10.0.1.1:7233 \ --node-key node-p2.key --listen 10.0.2.1:7233 … ``` With a pair per partition, give `--control` both kernels of the control partition, comma-separated. A partition serves no tenant before it has a map: two partitions must never take the same tenant. ## 3. Apply the map ```json { "epoch": 1, "partitions": { "p1": ["10.0.1.1:7233", "10.0.1.2:7233"], "p2": ["10.0.2.1:7233", "10.0.2.2:7233"] }, "default": "p1", "control": "p1", "tenants": { "acme": "p2" } } ``` ```sh kernable partitions apply 10.0.1.1:7233 ops.key map.json # ops.key: granted manage for * kernable partitions show 10.0.2.1:7233 ops.key # p2 has it within seconds ``` Tenants the map lists live on their partition; every other tenant on the default partition, so new tenants need no change to the map. Every new map needs a higher `epoch`. Keep the map file in version control: it is the answer to "where is tenant X?". Change a tenant's partition in the map only before it has data. A tenant with data moves with `kernable partitions move` (step 5); while a move runs, `partitions apply` refuses until it is finished. ## 4. Point your processes at it Give every process the address of one partition (or of all, comma-separated): the SDK reads the map there and - sends each submit, query, file and cache request to its tenant's partition; - runs its workers and listeners on every partition. A read model sees the events of all of them, in order within each partition; with a store, it keeps its position per partition; - applies memberships on the control partition; - reads the map again when a partition answers `wrong_partition` (the map changed), and sends the request anew. A subscription reads every partition too, from their beginnings, with events in order within a partition but not across them. A **read model across tenants** (totals over all customers, an admin list) holds the events of every partition, but a query reaches it through one partition, its tenant's, and the kernel there only knows how far the read model got on that partition. To get an answer that holds everything committed anywhere before the query, ask with `allPartitions`: ```ts [TypeScript] const total = await kernable.query({ domain: "billing", type: "Revenue", allPartitions: true }); ``` ```python [Python] total = k.query("billing", "Revenue", all_partitions=True) ``` ```rust [Rust] let total: u64 = kernable.query(Query::new("billing", "Revenue").all_partitions()).await?; ``` The SDK first sends a wait-only query to every partition at once; each returns as soon as the read model has processed that partition's events up to that moment, and then the query itself goes to the tenant's partition. It costs one round trip per partition, in parallel, plus however long the read model needs to catch up. Without partitions it waits for the one kernel the same way. ## 5. Move a tenant ```sh kernable partitions move 10.0.1.1:7233 ops.key acme p3 ``` The command moves `acme` from p2 to p3 in two parts, and applies a map for each: 1. **While `acme` runs on.** A map marks `acme` as being copied to p3. p3 asks p2 for every command of the tenant that has an outcome, with its events, vault values and files, and commits the events again under the same commands and worker signatures: new positions in p3's log, the same aggregate versions, and on each copy where it was first committed (`origin_partition`, `origin_position`). Listeners and subscriptions do not get the copies; the originals were delivered on p2. p3 holds none of `acme`'s unique keys yet. 2. **The rest, with `acme` stopped.** A map marks `acme` as moving. Both partitions answer its requests with `UNAVAILABLE` (`moving: …`); the SDKs wait and send them again, for up to 30 seconds. Once p2's queue holds none of `acme`'s commands, p2 sends only what p3 lacks (commands that completed since the first part) and what each of `acme`'s aggregates holds: its unique keys and facts. 3. p3 files those keys and tells the control partition, which hands it `acme`'s global unique keys. 4. The final map puts `acme` on p3. p2 drops `acme`'s unique keys, vault values and files; its events stay in p2's log as history. A listener (with the SDK) keeps a tenant's order across the move: it handles `acme`'s new events on p3 only after its stream of p2 has passed `acme`'s last event there. A subscription sees each partition in order, not across them. If the command stops (or a partition does), run it again: it continues the move, and p3 keeps what it copied already. Moving a tenant back works the same way; only what happened since it left is copied. The tenant is unavailable only for the second part. Measured on one Apple M2 Max with both partitions on it (`bench/move.sh`, a command per event, 1 KB each): with SQLite about a second, whatever the tenant's size (1 s for 268,000 events); with the file log, which reads its whole log for the rest, it grows with the log (2 s for 422,000 events, 1 GB). The move as a whole takes about 60 seconds per million events with SQLite and 30 with the file log, nearly all of it in the first part. Files go along in both parts, so a tenant with many large files stops longer. ## What it gains Every partition commits on its own, so throughput grows with partitions on machines of their own. Measured on one Apple M2 Max (12 CPUs, one SSD), where every partition also brings its worker and a load generator of 64 clients, so all of them share the same CPUs and disk (`bench/partitions.sh`, results in `bench/`): | Store, durability | 1 partition | 2 | 3 | 4 | |---|---|---|---|---| | SQLite, `normal` | 7,008/s | 10,279/s | 11,820/s | 12,287/s | | file log, `normal` | 12,670/s | 19,217/s | 20,909/s | 21,456/s | | SQLite, `full` | 3,052/s | 3,042/s | 3,675/s | 4,062/s | On one machine, two partitions give about 1.5 times one; beyond that the shared CPUs and disk are the limit, not kernable. Measure your own machines. ## What happens when | When | What happens | |---|---| | A request names a tenant of another partition | `FAILED_PRECONDITION` `wrong_partition: tenant acme is on partition p2 (map epoch 3)`; the SDKs read the map again and send it there | | A key is revoked on the control partition | Every partition refuses it within about a second | | The control partition is down | The others serve on with the map and trust records they hold; new maps, grants and memberships wait, and so do commits that file a new global key | | A partition that was down starts again | It takes what it missed from the control partition first thing | | A tenant moves | While it is copied it runs on; then its requests wait (`UNAVAILABLE`, `moving: …`) for the rest, about a second; the SDKs send them again | | The target of a move stops during the copy | It copies again when it is back; what it holds already it keeps | ## Limits - **One tenant never grows beyond one partition.** Give a big tenant a partition of its own. - **Across tenants, there is no order and no atomicity**: a command changes one aggregate of one tenant, as always; a read model across partitions sees each partition's events in order, not one global order. A query with `allPartitions` sees everything committed before it, but no order between partitions either. - **Unique keys with `scope: global`** are unique across all partitions: before a partition commits one its aggregate did not hold, it reserves the key on the control partition, which asks the partition that had it whether it still does (a moment of extra latency for those commits, and none while the control partition is down: they wait). Personal global keys are unique within each partition only. - New partitions in the map reach running workers and listeners when they start again. - A move copies the tenant's vault values only to a target with a vault. The source's standby drops the tenant's vault values and files only once it serves as the primary. --- # Monitor kernable Use this page to see what a running kernel is doing: in a browser while you develop or debug, and with health checks, metrics and alerts in production. To find the cause of a specific problem, start at [Failures and recovery](../concepts/failures#finding-the-cause). | Want to | Use | |---|---| | see at a glance whether anything needs attention | the UI's **Overview** (`--ui-listen`) | | see where the cases of a business process stand, and resolve its incidents | the UI's **Processes**, `kernable processes` | | see services, lag, parked events, rejections, the standby, the store | the UI's **Operations** | | see the domains, the log, one aggregate's history, or what the vault, snapshots, files and cache hold | the UI's **Explore** | | gate traffic and deploys | `GET /ready` (`--metrics-listen`) | | alert and graph | `GET /metrics` (`--metrics-listen`) | | list registrations with the kernel stopped or running | `kernable services`, `kernable workers`, `kernable listeners` | ## The UI With `--ui-listen` (on by default with `--dev`, at http://127.0.0.1:7234), the kernel shows itself in a browser. The start banner shows its address in the `UI` line, or `disabled`. The top bar has the four places below, a search box, and the **verdict**: *Healthy*, *Degraded* or the number of problems, with the reasons on hover. The page does not ask over and over: the kernel tells it when the log moved on or the verdict changed (server-sent events), and the page then reloads what it shows, at most every 2 s; without that connection it asks every 5 s. Every view has its own address (`/processes/shop.fulfilment`, `/processes/shop.fulfilment/cases/o-1` for the newest case of order `o-1`, `/explore/events?type=shop.Order`; another tenant than `default` as `?tenant=`), so a reload keeps it and a link can be sent to someone with the same tunnel. - **Overview:** commands and events over the kernel's run (up to a day, counted once a minute in memory: a restart starts over), cases started and open, incidents and rejected commands; a chart of activity per 10 minutes; **Needs attention**, every problem in one list with what fixes it (a listener missing or behind, parked events, incidents, a standby not in sync, a backup follower away, the store near its quota, a burst of rejections); the processes with their outcomes; and the system at a glance. - **Processes:** each [business process](../concepts/processes), one version at a time, over the last hour, day or week: - figures: cases started, ended per end with their share, the median and 95th percentile at the first wait, open cases and incidents; - the diagram, each path as wide as the number of cases that took it, with its count and share where a step branches, and the cases at each step now (incidents in red). It runs from left to right while it fits and from top to bottom when the process is too long; **Direction** sets it for good (kept in your browser). The conditions of the decisions are listed under it, and the ends have neutral colors, as no end is good or bad by itself. Click a step to list its cases; - outcomes over time, and the open cases by how long they have waited at their step, with the oldest one; - **Cases**, newest first, filtered by state, step and case, key or tenant; **Incidents** with the commands that resolve them (`kernable processes retry|skip|cancel`, ready to copy); and the statistics per step. A case has its own address (`/processes//cases/`, with `/` for an earlier case of the same key). It opens with an incident first, if it has one, with the commands that resolve it; then the diagram showing its own way (the steps it reached and how long it stayed at each, the rest faded) and as a story in plain words: what started it, what it waited for, what it sent and with what outcome, where it ended, each linked to the aggregates it touched; its raw events are one click away. The UI counts from the case events in the log, starting when you first open it: on a large log it says how far it has read until it has caught up. It keeps the running cases and the newest 1,000 ended ones per process, and its statistics for a week. - **Operations:** the kernel (version, uptime, queued and scheduled commands), the standby and witness, the backup follower, the store's size against `--max-store-bytes`, what limits refused; every worker and listener with its key, how far it was behind over the last hour and its parked events; **Parked events** with the command that delivers each again; and **Rejected commands** grouped by code, with why (the unique key taken, the requirement that did not hold). The process engine's own refused commands (a timer no longer needed) are left out unless you ask for them. Commands the kernel refuses before it accepts them (a bad signature, a missing grant) are not stored and do not show here; the kernel's log has them. - **Explore:** for developers. - **Domains:** totals over every domain (events in the log and in the last hour, commands in the last hour and the share rejected, open incidents), then one card per domain, by name or busiest first: what it holds (aggregates, commands, events, queries, processes), its numbers, its workers, and what needs attention (an aggregate without a worker, a reader away or behind, incidents in its processes), with links to its flow, its events and commands in the logs, and its part of the catalog. A domain opens with its numbers (events in the log, events and commands in the last hour with the share rejected, the last event), then per aggregate its commands (how many came in the last hour, how many were rejected, which processes send them) beside the events it writes (how many the log holds, the last hour per minute, the last one, who reads them); then the read models, listeners and middleware concerned with how far each is behind, its processes, and the events it uses from other domains. The UI counts the events the first time it opens a domain, from the start of the log, and from then on as they come. - **Catalog:** every event type, command, query and process, grouped by domain, in one table to filter by text or domain; a click on a column header sorts by it (then as one list, the domain before each name), a click on the first column back by domain and name. Events: who writes and who reads them, how many the log holds, the last hour per minute and the last one. Commands: how many came in the last hour and how many were rejected, who checks them and which processes send them; a domain whose aggregate has no worker says so. Queries: how many were asked in the last hour, how many were not answered (rejected, or no read model answered in time) and how long an answer took on average, the read model that answers, what it reads and how far it is behind; these counts start when the kernel starts. Processes: the event they start on, what they wait for, send and ask, their open cases and incidents, and how many started. A row opens the event log, the command log or the process. - **Event log:** the whole log, newest first under a heading per day, each event with who sent the command behind it (from the request context the event carries, no lookup needed), filtered by domain, event type, aggregate ID, command or correlation ID, text in the payload and tenant, without the process engine's events unless you include them. The filters in use show as chips to remove; the address keeps them. Events one command wrote together are marked on the left; the tenant column shows once more than one tenant is among them. **Live** shows new ones as they commit, highlighted for a moment. An event opens with its payload beside where it came from (sender, correlation, cause, case, command, worker key, hash) and links to the aggregate's history, the events of the same correlation, the command and the case. One search reads at most 20,000 events, so on a large log it may stop early and say how far back it got. - **Command log:** every stored command, newest first: when it was received, the aggregate, who sent it (a person, a service, or a process and its case), and what came of it: committed with the events it wrote and how long it took from receipt (or its due time) to the commit, rejected with the code and why, waiting, or scheduled for when. Commands come under a heading per day; those of one correlation next to each other are marked on the left. Filter by domain, type, aggregate ID, sender, command or correlation ID, outcome and tenant (the filters in use show as chips to remove, and the address keeps them); the process engine's own commands only on request; **Live** shows new ones, highlighted for a moment, and updates waiting ones. Open one for its payload beside who sent it and what came of it, with links to the aggregate's history, its events, the commands of the same correlation and the case. - **Data:** the stores beside the log, counted and listed but never a value: the UI does not decrypt the vault, show snapshot states or open files. - **Vault:** personal and secret values, how many subjects hold them and per aggregate type, the erasures (the vault keeps only a hash of the subject) and those not carried out yet, and how many values the vault key has encrypted against the 2³² one key may. Past half of that the verdict warns: plan a `kernable vault rekey` ([The vault's key](./keys#the-vault-s-key)). A subject is looked up by its exact ID: its values per aggregate type and its erasures. Subjects are not listed. - **Snapshots:** per aggregate type how many, their size and schemas, when the last was taken, and how many events the aggregates have gained since, with the aggregates furthest from their snapshot ([`snapshotEvery`](../reference/sdk-typescript#aggregateoptions)). - **Files:** per tenant and domain how many files and their size; a domain opens as folders by `/` in the keys, with each file's content type and size ([Store files and cache values](./data-and-cache)). - **Cache:** entries and memory against `--max-cache-bytes`, the hit rate, the values that expired or were evicted to make room, per domain, and the keys with how long until they expire. All of it starts with the kernel, as the cache does. - **Aggregates:** without one open, the aggregate types with how many there are and when one last changed; the list filters by type and by ID or tenant. An aggregate opens with its version, when it was created and last changed, its unique keys, facts and snapshot, the process cases it started, and links to its events and commands in the logs. Its events follow as a timeline under a heading per day, with the time between them, each with the command and sender that caused it and a color per correlation, so the events of one chain of commands share it; new ones appear live. One click opens the payload and the command beside where it came from and its signatures. Vault fields show as `personal · `, `secret` or `erased`; the UI never decrypts the vault, and it does not show snapshot states, which may hold personal values. - **Search:** an order, case, command or correlation ID finds its aggregates, its cases and its events. Keys show by their ID. Click one for its full ID (to copy), what it is used for now (the workers, listeners and processes it signs), and a name such as "app" or "operator" to show instead, kept in your browser only. The UI has no login, so it listens on a loopback address only; the kernel refuses to start with another address. Reach it on a server through an SSH tunnel: ```sh ssh -L 7234:127.0.0.1:7234 host # then open http://127.0.0.1:7234 ``` If the address is taken (another kernel in development), the kernel runs on without the UI and logs the warning `the UI is off`. ### Watching from a script The UI's own stream is open to anything on the machine, like the UI: server-sent events at `/api/watch`, read-only and without login, on the loopback address only. Choose what to get with `topics` and narrow it with filters: ```sh curl -N 'http://127.0.0.1:7234/api/watch?topics=events&domain=shop&type=order&id=o-1' # one order's events curl -N 'http://127.0.0.1:7234/api/watch?topics=commands&status=rejected' # rejected commands curl -N 'http://127.0.0.1:7234/api/watch?topics=cases&process=shop.fulfilment' # a process's cases curl -N 'http://127.0.0.1:7234/api/watch?topics=health,pulse' # the verdict, counts, listener positions ``` | Topic | Each message | |---|---| | `events` | the new events that match, with their sender and data (vault fields as references only) | | `commands` | commands that match when stored, and again when they complete: committed with their events and duration, rejected with why | | `cases` | the case events of a process (`process=`, `case=`) | | `head` | the log moved on, with the aggregates the new events changed | | `health` | the verdict, when it changed | | `pulse` | the head, the counts and each listener's position, when any changed | Filters: `tenant`, `domain`, `type` (aggregate type), `id` (aggregate ID), `event` (part of `domain.EventType`), `command` (command or correlation ID), `sender` (part of the identity), `process`, `case`, `status` (of commands); the process engine's own events and commands only with `engine=1`. Messages come at most once a second per topic. `events` and `cases` carry the log position as their id: connected again with `Last-Event-ID` (browsers do it by themselves) or `from=`, the stream goes on from there. More than 500 events behind, it sends `gap` instead: read what you missed from `/api/events`. For services elsewhere, subscribe through the SDKs instead: they are signed and checked against grants. ## Health checks Set `--metrics-listen` (for example `127.0.0.1:9233`). It serves plain HTTP with no login: bind it to a private address. On a loopback address it answers only requests whose `Host` is a loopback name (`localhost`, `127.0.0.1`, `[::1]`), as the UI does, so a web page cannot read it through DNS rebinding; health checks and Prometheus on the same machine send one anyway. - **Liveness:** `GET /health` answers 200 `ok` while the process runs. - **Readiness:** `GET /ready` answers 200 `ready` when the kernel can serve, and 503 with the reason otherwise: | 503 body | Meaning | |---|---| | `waiting for workers: shop.Cart` | no worker is connected for these aggregates | | `waiting for listeners: carts, guard` | these listeners or middlewares are not connected, or have not caught up yet | | `shutting down` | the kernel got SIGTERM and finishes running requests | It waits for what **counts**: what a service's membership lists, also members that never registered, and for services without a membership everything they registered. Dynamic members (registered, but not listed) and live listeners do not count ([Order and consistency](../concepts/ordering#who-counts)). Use it to gate traffic and deploys ([Deploy and upgrade](./deploy)). ## Alerts worth having ```yaml groups: - name: kernable rules: - alert: KernableNotReady # a worker or listener that counts is missing or behind expr: kernable_ready == 0 for: 2m - alert: KernableListenerLagging # a listener falls behind the log expr: kernable_listener_lag > 1000 for: 5m - alert: KernableListenerDown expr: kernable_listener_connected == 0 for: 2m - alert: KernableEventsParked # a reactor gave up on events: retry or dismiss them expr: kernable_listener_parked > 0 for: 1m - alert: KernableNoWorkers # commands queue up with nobody to run them expr: kernable_worker_sessions == 0 and kernable_queued_commands > 0 for: 1m - alert: KernableMiddlewareFailing # checks refused because a middleware is away or slow expr: rate(kernable_middleware_total{outcome="failed"}[5m]) > 0 for: 5m - alert: KernableQueriesFailing # no read model answered in time expr: rate(kernable_queries_total{outcome="failed"}[5m]) > 0 for: 5m - alert: KernableLimitReached # requests refused by a limit (--max-*) expr: rate(kernable_refused_total[5m]) > 0 for: 5m - alert: KernableStoreFilling # the stores passed 80 % of --max-store-bytes expr: kernable_store_bytes / kernable_store_bytes_max > 0.8 for: 15m ``` - `kernable_refused_total` names the limit in its `reason` label ([Limits](../reference/configuration#limits)). A growing `queued_commands` or `queued_per_aggregate` usually means a worker is away; `requests_per_key` a client that does not wait for its answers; `handshake_timeout` or `connections` something that opens connections and never uses them. `rate` is a key that starts more requests per second than `--max-requests-per-second-per-key`; `quota_storage` a kernel whose stores reached `--max-store-bytes`, which refuses new commands and file puts until data is deleted or the limit raised. - The per-listener metrics (`kernable_listener_lag`, `kernable_listener_connected`) cover every listener the kernel knows, also dynamic ones that nobody waits for. Exclude those by their `listener` label, or remove them ([Remove a listener](./remove-listener)). - A listener stuck at an event it cannot handle (a throwing handler, or a newer event version) shows as lag that keeps growing while the listener stays connected ([When a listener is slow or stuck](../concepts/ordering#when-a-listener-is-slow-or-stuck)). ## Useful views | Query | Shows | |---|---| | `sum by (outcome) (rate(kernable_commands_total[5m]))` | throughput and the share of rejections | | `rate(kernable_execution_seconds_sum[5m]) / rate(kernable_execution_seconds_count[5m])` | how long a command takes from lease to commit | | `rate(kernable_settle_seconds_sum[5m]) / rate(kernable_settle_seconds_count[5m])` | how long submits wait for listeners | | `rate(kernable_query_seconds_sum[5m]) / rate(kernable_query_seconds_count[5m])` | query latency through the kernel | | `rate(kernable_middleware_seconds_sum[5m]) / rate(kernable_middleware_seconds_count[5m])` | time spent in middleware checks | | `kernable_queued_commands` | commands waiting or running | The timings are summaries without quantiles: they give averages. All metrics are in the [reference](../reference/metrics). ## From the command line These read the store, whether the kernel runs or not; the UI shows the live state: ```sh kernable services sqlite:shop.db # each membership: who applied it, what it lists, (never registered), dynamic listeners kernable workers sqlite:shop.db # aggregate, command types, key, service, last seen kernable listeners sqlite:shop.db # name, order, position, types, key, service, middleware, parked, last seen kernable listeners parked sqlite:shop.db # each parked event: listener, position, type, code, attempts, when, message ``` `last seen` tells you when a worker or listener was last connected. See the [CLI reference](../reference/cli). ## Usage beside the store The kernel writes `.usage` beside its store (`kernable.db.usage`) every minute and when it stops, one line of JSON: ```json {"at":1793996832409,"events":205200,"commands":205211,"rejected":3,"queries":1200,"store_bytes":241172480,"sig":"9f3c…"} ``` `events` is the store's head, `store_bytes` what its stores take on the disk; `commands` (committed), `rejected` and `queries` (answered) go on across restarts: a kernel starts from the counts in the file, so a crash loses at most a minute of them. For whoever bills or watches the machine without scraping metrics. `sig` is the kernel key's signature (Ed25519, hex) over the other fields, one per line after the tag `kernable-usage-1`, in the order `at`, `events`, `commands`, `rejected`, `queries`, `store_bytes`, `replication` (empty without replication): checked with the kernel's public key, the counts are the kernel's, not another process's on the machine. ## Parked events A reactor that gives up on an event parks it and moves on ([Failures](../concepts/failures)): its follow-up was rejected, or its handler failed five times. The kernel keeps the event's position with the reason until you act on it; the UI shows the count on the listener, the first ones in its details and all of them under Connections, and `kernable_listener_parked` counts them. ```sh kernable listeners parked sqlite:shop.db # confirmations position 18204 shop.OrderSubmitted not_found attempts 1 parked 2026-11-02T14:03:00Z: no such customer # mailer position 18517 shop.OrderSubmitted handler_error attempts 5 parked 2026-11-02T14:21:07Z: TypeError ``` For a handler error the message is only the error's type; the reactor's log has the whole error, at the time it parked the event. Fix the cause, deploy, then deliver the events again, or drop those that no longer matter: ```sh kernable listeners retry 127.0.0.1:7233 ops.key mailer 18517 # or --all kernable listeners dismiss 127.0.0.1:7233 ops.key confirmations --all ``` The key needs `manage` for the listener's service. A retried event goes to the reactor on its own, before anything new, the next time it is connected; if it fails again, it is parked again. A follow-up command keeps its fixed ID, so retrying a rejected one returns the same rejection: retry once the handler deals with it differently, or dismiss it. ## Logs The kernel logs to stderr. `KERNABLE_LOG` sets the filter (default `info`; e.g. `debug` or `warn`) and `KERNABLE_LOG_FORMAT=json` switches to JSON for your log pipeline. The start banner appears only on a terminal, and not with JSON logs. Watch for `key revoked` and `drain timed out` warnings. The SDKs print what their workers and listeners do to stderr, one line each: ```text 12:04:10 worker connected to http://127.0.0.1:7233 · key app (69212a19f36f…) · runs shop.AddItem ``` There, `KERNABLE_LOG` is `off`, `error`, `warn`, `info` (default) or `debug`. Lines have colors on a terminal unless `NO_COLOR` is set. An app prints an overview of its workers and listeners once all are connected. --- # Mutual TLS with your own PKI Use this page when processes on other machines connect to the kernel. On a loopback address (`127.0.0.1:7233`, the default) or a Unix socket, the kernel speaks plain gRPC. On any other address it requires **mutual TLS**, and refuses to start without it: the kernel presents a certificate, and every process must present one signed by your client CA. TLS decides who may connect; [keys and grants](../concepts/trust#keys-and-grants) decide what they may do. You need both. ## Certificates | File | Requirements | |---|---| | the kernel's certificate | names the host your processes connect to, as a DNS name or an IP address in its subject alternative names; valid for server authentication; with its chain, if any | | client certificates | signed by the CA in `--tls-client-ca`; valid for client authentication. Any certificate of that CA is let in | | the CA for processes | the CA that signed the kernel's certificate | All files are PEM. For development, `kernable certs new` writes a CA and the certificates in one go: ```sh kernable certs new certs api shop-worker read-models # wrote ca.tls.crt, kernel.tls.crt/.tls.key and client certificates to certs ``` The kernel's certificate is valid for `localhost` and `127.0.0.1` only, and the CA key is not kept: to add a client, issue a new set. In production, use your own CA. ## 1. Start the kernel ```sh kernable start --root keys/root.pub --listen 0.0.0.0:7233 \ --tls-cert certs/kernel.crt \ --tls-key certs/kernel.key \ --tls-client-ca certs/clients-ca.crt ``` The three flags go together, and apply to TCP only; a kernel on a Unix socket refuses them. Each has an environment variable (`KERNABLE_KERNEL_TLS_CERT`, `KERNABLE_KERNEL_TLS_KEY`, `KERNABLE_TLS_CLIENT_CA`), see [kernable start](../reference/configuration). The start banner shows `Listen ... 0.0.0.0:7233 (mutual TLS)`. Without them, a non-loopback address fails at start: `0.0.0.0:7233 is reachable from other machines: plain TCP only on a loopback address (127.0.0.1:7233); elsewhere, use mutual TLS (--tls-cert, --tls-key, --tls-client-ca)`. ## 2. Configure your processes Give every process the CA that signed the kernel's certificate, and its own certificate and key. Either set three environment variables: ```sh export KERNABLE_URL=https://kernel.internal:7233 export KERNABLE_TLS_CA=certs/kernel-ca.crt export KERNABLE_TLS_CERT=certs/shop-worker.crt export KERNABLE_TLS_KEY=certs/shop-worker.key ``` or pass them in code: | SDK | Option | |---|---| | TypeScript | `new Kernable({ tls: { ca, cert, key } })` | | Python | `Kernable(tls=Tls(ca, cert, key))` | | Rust | `Options::new("shop-worker.key").tls(Tls { ca, cert, key })` (paths) | Set all three variables or none: in the SDKs, one or two of them is an error. Without TLS, the SDKs refuse a kernel that is not on the same machine (anything but a Unix socket or a loopback address): grants, signed requests and vault values would cross the network readable. To allow it anyway, for example through a tunnel that encrypts on its own, set `KERNABLE_INSECURE=1` or the option `insecure` (TypeScript, Python) or `Options::insecure()` (Rust). The CLI commands that talk to a running kernel (`kernable apply`, `kernable revoke`, `kernable submit`) read the same variables, but use TLS only when all three are set; with one or two, they quietly connect without TLS, and the kernel refuses them on a non-loopback address. ## Pitfalls - **Connect by a name in the certificate.** The host in the URL (`kernel.internal`) is checked against the kernel certificate's subject alternative names. A certificate for `kernel.internal` fails when you connect by IP address, and the reverse. - **`http://` or `https://` makes no difference.** With TLS set, the SDKs and the CLI connect over TLS either way. - **The metrics and UI addresses are not covered.** `--metrics-listen` serves plain HTTP: bind it to a private address. `--ui-listen` is loopback only ([Monitor kernable](./monitor)). ## Renewing certificates Replace the files and restart: the kernel reads its certificate at start, processes when they connect. Processes reconnect on their own after the kernel's restart. --- # Keys per service, and changing a key Use this page when you move from the tutorial's single key to production, give a new service its rights, or replace a key: routinely, or because it leaked. How keys and grants work is in [Trust](../concepts/trust#keys-and-grants). | Situation | What to do | Kernel restart? | |---|---|---| | a new service, or more rights | [issue a grant](#one-key-per-service) | no | | fewer rights | [change the key](#changing-a-key), revoke the old one | no | | a key leaked | [revoke it](#revoking-a-stolen-key) | no | | the kernel's key | [replace the file, SIGHUP](#the-kernel-s-key) | no | | the vault's key | [re-encrypt the vault](#the-vault-s-key) | yes | | the root key | [start with both roots](#the-root-key) | yes, twice | ## One key per service The tutorial uses one key for everything (`--dev` writes `app.key` with a grant for everything). In production, give every service its own key and grant it only what it needs. A root key signs the grants; the kernel trusts root keys only. 1. **Create the keys.** Each call writes `.key` (secret) and `.pub` (the key ID): ```sh for k in root api confirmations shop-worker read-models deploy; do kernable keys new keys/$k; done ``` Keep `keys/root.key` offline; the kernel needs only `keys/root.pub`. 2. **Grant each key what it needs:** ```sh kernable grant keys/root.key keys/api.pub \ submit=shop.Register,shop.Place,shop.AddItem,shop.Submit 'query=shop.*' kernable grant keys/root.key keys/confirmations.pub \ submit=shop.Confirm listen=shop.OrderSubmitted query=shop.GetOrder,shop.GetCustomer kernable grant keys/root.key keys/shop-worker.pub 'run=shop.*' kernable grant keys/root.key keys/read-models.pub 'listen=shop.*' 'answer=shop.*' 'vault_read=shop.*' kernable grant keys/root.key keys/deploy.pub manage=shop # applies the shop's membership ``` Each call prints the permissions and `wrote keys/.grant`. A permission is `=,`, or a bare type for every action; a type is `shop.Place`, `shop.*` or `*`. The actions are in [Trust](../concepts/trust#keys-and-grants); `manage` takes service names. 3. **Start the kernel with the root's public key:** `kernable start --root keys/root.pub ...`. 4. **Give each service its `.key` and `.grant` file.** The SDKs read the `.grant` file next to the key file, or the file in the `grant` option or `KERNABLE_GRANT`, and send it with every call. The key file comes from the `key` option or `KERNABLE_KEY` (default `app.key`). A process that runs an app (`kernable.app(...).run()`) applies its service's membership on every start, so its key also needs `manage=` ([Deploy and upgrade](./deploy)). Without that, a single deploy key applies it. A key without a grant, or whose grant does not cover a type, gets `PERMISSION_DENIED`. A grant takes effect when a process first presents it: the kernel's settings hold no keys of services, so a new service needs no change to the kernel. ## Changing what a key may do Issue a new grant for the key and restart the service with it. The kernel keeps the newest grant per key: once it has seen the new one, it refuses the old one, and workers, listeners and subscriptions whose key may no longer do what it does are disconnected. A listener or subscription that asked for every event type stays connected and gets only the types the new grant allows. A new service needs only its grant; the kernel does not change. A grant cannot be withdrawn, only replaced by a newer one, which the kernel keeps across restarts. To take rights away from a key that may be stolen, change the key and revoke the old one. ## Changing a key Signatures in the log stay verifiable after a change: a key ID is the public key itself. **A client or worker key:** create the new key, grant it what the old one had, and restart the process with the new key and grant. Revoke the old key if it may have leaked. **A listener's key** is part of its registration. Grant the new key with `--succeeds `, and the listener keeps its position, order and middleware registration: ```sh kernable keys new keys/read-models-2 kernable grant keys/root.key keys/read-models-2.pub --succeeds keys/read-models.pub \ 'listen=shop.*' 'answer=shop.*' 'vault_read=shop.*' ``` Then restart the listener with the new key and grant. Without `succeeds`, another key is refused, so no service can take over another's listener. ## Revoking a stolen key When a key leaks, revoke it from the moment it may have been used. The root key signs the revocation, and the running kernel takes it at once: ```sh kernable revoke http://127.0.0.1:7233 keys/root.key keys/api.pub 2026-10-27T09:00:00Z # UTC ``` It answers `revoked `. `kernable revoke` connects over mutual TLS when `KERNABLE_TLS_CA`, `KERNABLE_TLS_CERT` and `KERNABLE_TLS_KEY` are set, like the SDKs. The revoke request is valid for 60 s around its creation. The kernel then: - stores the revocation, signed with its own key; it is never undone and holds after a restart; - refuses the key everywhere, also with a grant, and disconnects its sessions; - rejects commands of the key that still wait in the queue and were accepted after that time (`key_revoked`). `kernable verify` then lists everything the key signed from that time on: ```text $ kernable verify sqlite:data/shop.db keys/kernel.pub 2311 events, 1903 commands, 41 checkpoints, 0 events after the last checkpoint SUSPECT command c-8f2a: signed by revoked key 1f0a… OK, 1 suspect ``` It exits with 2 when there are suspects, 1 when the log was tampered with. Review what the suspects did, and undo it with new commands where needed: the log itself is never rewritten. ## The kernel's key The kernel's key signs checkpoints and revocations. Replace it without a restart: write the new key to the file `--kernel-key` names, and send the kernel SIGHUP (Unix only). `kernable keys new` never overwrites a key, so create it under another name and move it: ```sh cp keys/kernel.pub keys/kernel-old.pub # verify needs it later kernable keys new keys/kernel-2 mv keys/kernel-2.key keys/kernel.key && mv keys/kernel-2.pub keys/kernel.pub kill -HUP ``` The kernel logs `kernel key read again`, and signs with the new key from then on. If it cannot read the file, it logs an error and keeps the previous key. SIGHUP changes nothing else; other settings need a restart. Keep every old public key: `kernable verify` needs every key the kernel ever signed with, `kernable verify keys/kernel-old.pub keys/kernel.pub`. ## The vault's key Stop the kernel, encrypt the vault again, and start it with `--vault-key` pointing to the new file: ```sh kernable keys new keys/vault-2 >/dev/null kernable vault rekey sqlite:data/vault.db keys/vault.key keys/vault-2.key ``` The old key cannot read the vault afterwards. Backups made before still need the old key. Rekey also when a key has encrypted many values. Each value gets a random nonce, and one key is safe for 2^32 of them. The kernel counts them, logs a warning at startup past 2^31 (the UI's verdict warns too, and **Explore → Data → Vault** shows the count), and refuses new values at 2^32 until the vault is rekeyed. ## Keys that are not on the disk in clear With `--key-command`, the key files of the kernel, the vault and the node hold **wrapped** keys, and a program you name unwraps them when the kernel starts (and on SIGHUP). The kernel runs it with the key file's path as its last argument and reads the key from its output: 64 hex characters, as a key file holds them. Nothing in the kernel depends on a provider; any tool that decrypts a file works. With [age](https://age-encryption.org) and an identity on a hardware key or in a protected place: ```sh age -e -r age1… -o kernel.key.age kernel.key && shred -u kernel.key kernable start --kernel-key kernel.key.age --key-command "age -d -i /etc/kernable/age.identity" … ``` With systemd-creds, bound to the machine's TPM: ```sh systemd-creds encrypt --with-key=tpm2 kernel.key kernel.key.cred && shred -u kernel.key kernable start --kernel-key kernel.key.cred --key-command "systemd-creds decrypt" … ``` With a cloud KMS, a small script that takes the path, for example `/usr/local/bin/kms-unwrap`: ```sh #!/bin/sh aws kms decrypt --ciphertext-blob "fileb://$1" --query Plaintext --output text | base64 -d ``` The command runs once per key file and has 30 s for it; the kernel keeps the unwrapped key in memory until SIGHUP, when it runs the command again. A key the command cannot unwrap stops the kernel at start, with the command's message; on SIGHUP the previous key stays. A key bound to one machine (the TPM) needs wrapping again for the standby on another one. TLS keys (`--tls-key`) are read as files. ## The root key The root key signs only grants and revocations, so it can stay offline: issue grants on a separate machine and copy the `.grant` files. To replace it, start the kernel with both roots (`--root keys/root.pub --root keys/root-2.pub`), issue new grants with the new root and restart the services with them. Then restart the kernel without the old `--root`: grants signed by it are refused from then on. If the old root leaked, revoke it first, signed with the new root (`kernable revoke keys/root-2.key keys/root.pub `): the kernel drops the grants the old root signed at once and refuses them from then on. --- # Change an event's format Use this page when an event must carry something different: a field renamed, removed, split, or of another type. Stored events are never rewritten. You raise the event's version and tell every reader how to turn the old payload into the new one ([Event versions](../concepts/data#event-versions)). ## Steps 1. **Declare the new version and an upcast** on the domain: ::: code-group TypeScript: ```ts export const shopDomain = domain("shop"); shopDomain.event("ItemAdded", 2).upcast(1, (d: { sku: string; qty: number }) => ({ sku: d.sku, count: d.qty })); ``` ```python [Python] shop_domain = domain("shop") shop_domain.event("ItemAdded", 2).upcast(1, lambda d: {"sku": d["sku"], "count": d["qty"]}) ``` ```rust [Rust] let shop_domain = domain("shop"); shop_domain .event("ItemAdded", 2) .upcast(1, |d| json!({ "sku": d["sku"], "count": d["qty"] })); ``` ::: One upcast per old version: `upcast(1, …)` turns 1 into 2, `upcast(2, …)` turns 2 into 3, and they chain. The domain's aggregates and listeners use them before `apply` and before their handlers. A listener of **another** domain that reads the event needs the domain too: `use(shopDomain)` (TypeScript, Python) or `with_domain(&shop_domain)` (Rust). 2. **Change the readers**: `apply` of the aggregate, and every read model, reactor and middleware that handles the event, now get the new payload only. 3. **Write the new format** in the command handlers. 4. **Deploy the readers first**, then the workers. Every process that reads the event needs the new domain definition, in whatever language it is written. 5. **Check** with `kernable schemas `. It prints the highest stored version per event type, for example `shop.ItemAdded schema 2`. ## What happens during the deploy - **A reader that meets a version it does not know stops** at that event instead of misreading it. It logs `shop.ItemAdded has schema 2, but this process knows it only up to 1: update it`, retries, and keeps its position. Listeners with a higher order wait behind it. Its lag grows in the metrics and in the UI ([Monitor](./monitor)). It continues once the process is updated. - **A worker cannot write an older version** than the store holds. Once a version-2 event is stored, the kernel refuses an old worker's result, ends its session and puts the command back in the queue for a current worker. Old and new workers never write side by side. If readers and writers run in one app, deploy it as usual: old processes still running stop at the first new event and continue as soon as they are replaced. ## Pitfalls - **A listener of another domain without `use`** knows the event only at version 1, so it stops at the first version-2 event. - **State shapes are separate.** If the change also changes an aggregate's state, raise the aggregate's `schema` so old snapshots are rebuilt. If it changes what a read model with a store keeps, raise the read model's `schema` ([Rebuild a read model](./rebuild-read-model)). - **Avoid a new version** when you can: a new optional field that every reader can ignore or default needs none. **Use a new event type** when the meaning changes, not only the shape. Archives and exports that must see events exactly as stored mark their handler `raw` (`{ raw: true }` in TypeScript, `raw=True` in Python, `on_raw` in Rust). They get every version unchanged, with `event.schema`. --- # Rebuild a read model A read model is derived from the events, so you can always build it again. Rebuild one after fixing a bug in a handler, when it should hold something new, or when its stored data changes shape. ## How It depends on where the read model keeps its data ([Snapshots and read model stores](../concepts/data#snapshots-and-read-model-stores)): - **In memory** (no `store`, the default): deploy the fixed version, or restart it. A read model starts at position 0 by default (`start: 0`), so it reads every event from the first on every start. - **With a store** (`store: "sqlite:carts.db"`): a restart continues after the stored position, so a fix does not reach old data by itself. Raise its `schema` (`schema: "2"`; `schema="2"` in Python, `.store(url, "2")` in Rust). On start it clears the store and reads every event from the first. Raise it whenever the handlers store data differently, too. Or stop the process and delete the store file, with its `-wal` and `-shm` files. It starts empty, at position 0. ## While it rebuilds It is **behind** until it reaches the events that were stored when it connected: - queries with `minPosition` wait for it; - submits that settle wait for it, up to `--settle-timeout-ms` (default 5 s), then throw `NotSettled`; - listeners with a higher order, such as reactors, wait behind it; - `/ready` answers 503 with `waiting for listeners: carts`, and the kernel UI shows it as behind, with its lag ([Monitor](./monitor)). When `/ready` answers 200 again, it has caught up. For a large log, rebuild outside peak hours, or **rebuild beside the old one**: 1. Run the new version under a **new name** (and, with a store, a new store file) next to the old one. 2. Wait until it caught up: `kernable_listener_lag` for it is near 0. 3. Switch the queries over to the new name. 4. Remove the old one ([Remove a listener, middleware or aggregate](./remove-listener)). If the service has a membership that does not list the new name yet, the new read model is dynamic while it rebuilds: nobody waits for it. Add it to the membership when you switch over. `app()` lists every read model of its domains, so there the new one counts from its first start. ## Reactors are not rebuilt A reactor acts on events: reading every event again would act again, send every mail again. A reactor continues after its last acknowledged event, and a new one starts at the newest event. So never give a reactor `start: 0`, and never raise the `schema` of a reactor's store or delete its store file: each makes it read every event from the first. --- # Remove a listener, middleware or aggregate Use this page when you delete or rename a read model, reactor, middleware or aggregate. Taking it out of the code is not enough: the kernel remembers every listener that ever connected, with its position, and every aggregate a worker registered. One that stops running still **counts** ([Who counts](../concepts/ordering#who-counts)): - listeners with a higher `order` get no events past its position; - submits that settle wait for it, until `--settle-timeout-ms`, then throw `NotSettled`; - `/ready` answers 503 (`waiting for listeners: old-orders`); - if it was a middleware, everything it checked is refused; - if it was an aggregate, its commands wait in the queue for a worker that never comes. ## Steps 1. **Delete it from the code** and deploy. 2. **Apply the service's membership without it** (below). With `app()`, step 1 already does this: the new version applies a membership built from its domains when it starts. 3. **Check** with `kernable services `, or in the kernel UI. The service no longer lists it, and `/ready` answers 200. 4. Optionally, **remove it for good** (below), so it no longer shows up and its name is free. ## Apply the service's membership Workers and listeners register themselves whenever they connect. Which of them the kernel waits for is a separate, signed list: the service's **membership**. Your deploy applies it, once per deploy: ```ts // apply.ts, run by the deploy before or after starting the new version const kernable = new Kernable({ service: "shop" }); // or KERNABLE_SERVICE=shop await kernable.apply({ workers: [cartAggregate], listeners: [cartsReadmodel, mailsReactor], middlewares: [guardReadmodel], metadata: { version: "1.4.0" }, }); ``` If the service is simply made of whole domains, let the SDK build the membership: `app()` lists every aggregate, read model and reactor defined on the domains (live ones excepted, and those that check requests as middlewares), applies it, and runs all of them: ```ts await new Kernable().app({ service: "shop", version: "1.4.0", domains: [shopDomain, billingDomain] }).run(); ``` One service is one app: two different processes with the same service would replace each other's membership. Or from the pipeline, with a key granted `manage=shop`: ```sh kernable apply 127.0.0.1:7233 deploy.key shop.json # shop.json: {"service": "shop", "workers": ["shop.cart"], "listeners": ["carts", "mails"], # "middlewares": ["guard"], "metadata": {"version": "1.4.0"}} ``` Workers are named `domain.aggregate`, listeners and middlewares by their name. The processes of the service name it too (`service` option or `KERNABLE_SERVICE`). Then: - **Listed** workers and listeners count, also before they ever connected: `/ready` waits for them, and commands of a listed aggregate wait in the queue until its worker comes. - **Not listed** ones of the service are **dynamic**. They run, but nobody waits for them. A dynamic listener keeps its stored position; a dynamic aggregate is forgotten once its last worker leaves, and its commands are refused from then on. - **Middlewares** must be listed as such: a listener of the service that is not listed under `middlewares` cannot connect as one. Taking a middleware off the list removes its check. A check never appears or disappears without a signed membership, so a missing process never switches security off: while a listed middleware is away, what it checks is refused. Apply it **once per deploy**, not from every process: the newest apply wins, and an old replica that restarted during a rolling deploy would otherwise bring the old list back. Processes without a service, and services without a membership, keep the simple rule: everything registered counts. To remove one of those, remove it for good. `kernable services ` shows each service's membership, what it lists that never registered, and its dynamic listeners: ``` shop version=1.4.0 (applied … by …) worker shop.cart listener carts listener mails middleware guard dynamic listener old-orders ``` ## Remove it for good A dynamic listener no longer holds anything back, but the kernel still keeps its name and position. If you later start a reactor under the same name, it continues from that old position. To forget it, stop the kernel and run: ```sh kernable listeners sqlite:shop.db # name, order, position, types, service, middleware, last seen kernable listeners remove sqlite:shop.db old-orders kernable workers sqlite:shop.db # aggregate, command types, key, service, last seen kernable workers remove sqlite:shop.db shop.wishlist ``` `last seen` tells you when a listener, or a worker of an aggregate, was last connected; a listener its service no longer lists shows `(not listed: counts for nobody)`. Then start the kernel. Removing a listener also removes its middleware. Commands for a removed aggregate are refused. ## Listeners that need no registration A **live** listener (`live: true`) is never registered: it has no stored position, starts at the newest event, nobody waits for it, and the kernel forgets it when it disconnects. Use it for reactions that only matter while the process runs, such as pushing updates to connected browsers. There is nothing to remove. It cannot be a middleware. --- # Test your domain Aggregates are plain functions of events and commands ([Decide and fold](../concepts/event-sourcing#decide-and-fold)), so most of your domain can be tested without a kernel: fold past events into a state with the aggregate's `fold`, then call a command handler on that state and check the events it returns, or the rejection. ## Aggregates, without a kernel order.test.ts: ```ts /** Runs `type` on the state after `history`, as a worker would. */ function decide(history: { type: string; data?: unknown }[], type: string, data: unknown = {}) { const state = orderAggregate.fold(history.map((e) => ({ schema: 1, data: {}, ...e }))); return orderAggregate.handle(state, type, data, new CommandContext("test", {}, new Date("2026-01-01T00:00:00Z"))); } test("a draft order with items can be submitted", async () => { const placed = { type: "OrderPlaced", data: { customerId: "c1" } }; const added = { type: "ItemAdded", data: { sku: "apple", qty: 2 } }; assert.deepEqual(await decide([placed, added], "Submit"), [{ type: "OrderSubmitted" }]); }); test("an empty order cannot be submitted", async () => { const placed = { type: "OrderPlaced", data: { customerId: "c1" } }; await assert.rejects(async () => decide([placed], "Submit"), (err: Rejected) => err.code === "invalid_state"); }); ``` test_order.py: ```py def decide(history, type, data=None): """Runs `type` on the state after `history`, as a worker would.""" state = order_aggregate.fold([(e["type"], 1, e.get("data", {})) for e in history]) at = datetime(2026, 1, 1, tzinfo=timezone.utc) return order_aggregate.handle(state, type, data, CommandContext.new("test", received_at=at)) class OrderTest(unittest.TestCase): placed = {"type": "OrderPlaced", "data": {"customerId": "c1"}} def test_a_draft_order_with_items_can_be_submitted(self): added = {"type": "ItemAdded", "data": {"sku": "apple", "qty": 2}} self.assertEqual(decide([self.placed, added], "Submit"), [{"type": "OrderSubmitted"}]) def test_an_empty_order_cannot_be_submitted(self): with self.assertRaises(Rejected) as rejected: decide([self.placed], "Submit") self.assertEqual(rejected.exception.code, "invalid_state") ``` tests/order.rs: ```rs /// Runs `command_type` on the state after `history`, as a worker would. fn decide(history: &[(&str, Value)], command_type: &str) -> Result, Error> { let order_aggregate = shop::shop().order_aggregate; let state = order_aggregate.fold( history.iter().map(|(t, data)| FoldEvent::new(t, data)), None, )?; let mut ctx = CommandContext::new( "test", RequestContext::default(), Timestamp(1_767_225_600_000), ); order_aggregate.handle(&state, command_type, (), &mut ctx) } #[test] fn a_draft_order_with_items_can_be_submitted() { let placed = ("OrderPlaced", json!({ "customerId": "c1" })); let added = ("ItemAdded", json!({ "sku": "apple", "qty": 2 })); assert_eq!( decide(&[placed, added], "Submit").unwrap(), [NewEvent::new("OrderSubmitted", ())] ); } #[test] fn an_empty_order_cannot_be_submitted() { let placed = ("OrderPlaced", json!({ "customerId": "c1" })); assert!( matches!(decide(&[placed], "Submit"), Err(Error::Rejected(r)) if r.code == "invalid_state") ); } ``` These run with `node --test`, `python -m unittest` and `cargo test`. They test the [tutorial's](../tutorial/shop/2-commands) `order` aggregate. - **Given** is the history you fold: events as `{ type, schema, data }` in TypeScript, `(type, schema, data)` tuples in Python, `FoldEvent::new(type, data)` in Rust (schema 1). - **When** is the handler you call, with a fixed clock and fixed IDs in the context, so the result is the same on every run. - **Then** is the list of new events, or a `Rejected` with its `code`. `fold` upcasts every event first, as a worker does. To test an upcast ([Change an event's format](./event-versions)), fold an event at its old schema, such as `{ type: "ItemAdded", schema: 1, data: { sku: "apple", qty: 2 } }`, and check the state. To test unique keys and facts, pass a folded state to the aggregate's `index`: it returns the keys and facts a worker would send for it ([Unique keys and conditions](../concepts/uniqueness#testing)). ## Read models, reactors and middleware, against a real kernel These depend on the kernel: order, settling, queries and checks. Test them end to end: 1. Start a kernel that keeps nothing, in a temporary directory: ```sh cd "$(mktemp -d)" kernable start --dev --store memory: --listen 127.0.0.1:17233 --metrics-listen 127.0.0.1:17234 ``` `--dev` creates the keys there, among them `app.key` with a grant that may do everything. 2. Start your processes against it, with `KERNABLE_URL=http://127.0.0.1:17233` and `KERNABLE_KEY=/app.key`. 3. Wait until they are connected: `await app.ready` in TypeScript, `app.ready.wait(5)` in Python, `.ready().await` in Rust. Do not sleep. 4. Send commands and queries as your API would. A submit settles by default, so the next query already sees its events. Do not rely on `/ready` alone to wait for your processes: before they connect for the first time, the kernel does not know them, unless their service's membership was applied. To see how kernable sets this up for its own tests, look at `sdks/typescript/test/kernel.ts`, `sdks/python/tests/kernel.py` and `sdks/rust/tests/common/mod.rs` in the repository. These helpers are not part of the SDKs. --- # Schedule a command Some commands should run later: cancel an order that is not paid within 3 days, remind a customer tomorrow, close a poll at midnight. Enqueue the command now with the time it should run at; the kernel keeps it and runs it then. ```ts [TypeScript] const id = await kernable.enqueue(CancelOrder.new({ reason: "unpaid" }), { aggregateId: "o1", at: new Date(Date.now() + 3 * 24 * 60 * 60 * 1000), // in 3 days }); ``` ```python [Python] from datetime import datetime, timedelta, timezone command_id = kernable.enqueue( CancelOrder(reason="unpaid"), aggregate_id="o1", at=datetime.now(timezone.utc) + timedelta(days=3), ) ``` ```rust [Rust] use std::time::{Duration, SystemTime}; let id = kernable .enqueue( Submit::command(CancelOrder { reason: "unpaid".into() }, "o1") .at(SystemTime::now() + Duration::from_secs(3 * 24 * 60 * 60)), // in 3 days ) .await?; ``` `enqueue` returns at once with the command's ID. The time is part of the signed command. ## What happens until then - **The command is stored now.** Like any queued command it survives restarts, reaches the standby before you hear back, and moves with its tenant to another partition. - **The aggregate is not blocked.** Commands for an aggregate run one at a time, in order; a scheduled command waits beside that queue, so the order's other commands run meanwhile. At its time it joins the queue behind them. - **It is checked twice.** When you enqueue it: the signature, the key's grant and the [middleware](../concepts/queries-and-middleware#middleware), so a command nobody may send is refused at once. When it is due: the middleware again, with what it knows then, and whether its key was revoked meanwhile. A customer suspended in the meantime does not get the command run. - **Late runs late.** If the kernel is down when the command is due, it runs as soon as the kernel is back. A command that must not run late takes `until` as well: past that time it is rejected `expired` instead. - **At most `--schedule-max-days` ahead** (366 by default). ## Cancel it When the order is paid after a day, the cancellation must not happen: ```ts [TypeScript] const result = await kernable.cancel(id); // { cancelled: true }, or { cancelled: false, code: "too_late" | "not_found" | , message } ``` ```python [Python] result = kernable.cancel(command_id) # CancelOutcome(cancelled=True), or cancelled=False with code "too_late", "not_found" or a middleware's ``` ```rust [Rust] let outcome = kernable.cancel(Cancel::new(&id)).await?; // CancelOutcome::Cancelled, or CancelOutcome::Rejected(r) with r.code "too_late", "not_found" or a middleware's ``` - **Who may cancel:** a key granted `submit` for the command's type, the same as could send it. Middlewares check the cancel as a request of its own kind (`check.kind` is `"cancel"`), with the command's type, aggregate and data. The [kit](../kits/authorization) decides a cancel exactly as it would decide the command. - **Afterwards** the command's ID answers `cancelled`, also to a resubmit. - **Too late:** a command that is due already, running or done answers `too_late`. Undo its effect with another command if you need to. ## Its result Submit the same command ID again to get the outcome. Before the command's time, this waits for it, up to the kernel's submit timeout; afterwards it returns the stored outcome at once, as for any command. ## Watch it `kernable_scheduled_commands` counts the scheduled commands waiting. Cancelled and expired ones count as rejected in `kernable_commands_total`, and appear in the UI's rejected commands (Operations) with their code. --- # Handle an erasure request A person asks you to erase their personal data, for example under the GDPR. Events cannot be deleted, so kernable keeps personal values in the **vault** and the log keeps references only ([Personal data: the vault](../concepts/data#personal-data-the-vault)). With that set up, an erasure is one command. The set-up has to be in place **before** the data arrives: a value that went into the log in clear stays there forever. ## Before the first request: declare 1. **Turn the vault on:** start the kernel with `--vault-key` (`kernable start --dev` does). Without it, commands and events with vault fields are refused. 2. **Mark every personal field**, in commands and events, with the person it belongs to (`personal(fields, subject)`; [tutorial, part 7](../tutorial/shop/7-personal-data)). Fields of other aggregates and domains count too: the shop marks the email in an order's confirmation as personal data of the customer. Only top-level fields can be marked. A worker rejects a command (`personal_in_clear`) whose handler puts a field the command marks personal, under the same name, into an event that does not: it would be stored in clear. After an erasure, and for a reader without `vault_read`, such a field is `null`: in TypeScript, type it `Personal` so the compiler finds the handlers that must allow for that. 3. **Declare an erasing event**, e.g. `CustomerErased` with `erases(subject)`, and a command that writes it, e.g. `Erase` on the customer. 4. **Grant the worker** that writes it `vault_erase` for that event type: ```sh kernable grant keys/root.key keys/shop-worker.pub 'run=shop.*' vault_erase=shop.CustomerErased ``` Without it, the command is rejected with `erase_not_allowed`. 5. **Make every read model that holds personal data forget the person** when it sees the erasing event. A read model's store then also removes the old values from its write-ahead log, so they are not left on the disk. Do the same for files: an erasure does not touch them ([Personal data in files](./data-and-cache#personal-data-in-files)). 6. **Let clients seal the fields:** a client that sends commands with personal fields needs the domain's declarations (`.use(shopDomain)`, Python `.use(shop_domain)`, Rust `.with_domain(&shop_domain)`). Without them, it sends the values in clear and they are stored in the command log. The shop's read models forget an erased customer like this: read-models.ts: ```ts // An erased customer's personal data must not outlive the erasure in a read model either. customersReadmodel.on("CustomerErased", (_data, evt) => void customers.delete(evt.aggregateId)); ordersReadmodel.on("CustomerErased", (_data, evt) => { for (const row of orders.values()) if (row.customerId === evt.aggregateId && row.confirmedTo) row.confirmedTo = null; }); ``` read_models.py: ```py # An erased customer's personal data must not outlive the erasure in a read model either. @customers_readmodel.on("CustomerErased") def _(data, evt): customers.pop(evt.aggregate_id, None) @orders_readmodel.on("CustomerErased") def _(data, evt): for row in orders.values(): if row["customerId"] == evt.aggregate_id and row.get("confirmedTo"): row["confirmedTo"] = None ``` read_models.rs: ```rs // An erased customer's personal data must not outlive the erasure in a read model either. let rows = Arc::clone(&customers); customers_readmodel.on("CustomerErased", move |_: Value, evt, _ctx| { rows.lock().unwrap().remove(&evt.aggregate_id); async { Ok(()) } }); let rows = Arc::clone(&orders); orders_readmodel.on("CustomerErased", move |_: Value, evt, _ctx| { for row in rows.lock().unwrap().values_mut() { if row.customer_id == evt.aggregate_id && row.confirmed_to.is_some() { row.confirmed_to = Some(None); } } async { Ok(()) } }); ``` The subject is a string, such as the customer ID `c1`. An erasure deletes every value of that subject in the erasing command's tenant, in every stream and domain; the same subject in another tenant keeps its values. Use one subject per person, unique within the tenant. ## The request 1. **Authenticate** the person, as for any request. 2. **Send the erasing command:** ```ts const outcome = await kernable.submit({ domain: "shop", aggregateType: "customer", aggregateId: "c1", type: "Erase" }); ``` When it is committed, the kernel has deleted the person's values from the vault, in every stream and domain of its tenant, and dropped the snapshots of every stream that used them. Unique keys built from the person's data are free again ([personal keys](../concepts/uniqueness#personal-keys)). Secrets have no subject and are not affected. The kernel stores the erasing event first and erases right after, before it answers; after a crash in between, it finishes the erasure when it starts, before it serves anything. So the log records every erasure that happened, and a rejected erasing command erases nothing. 3. **Read models** forget the person as they process the event; rebuilt, they read `null` where the values were. 4. **Delete the person's files** if a reactor does not do it for you. 5. **Backups** of the vault still hold the values until they expire ([Back up and restore](./backup#backups-and-erasure)). 6. **Confirm** to the person. ## Check that it worked The kernel logs `erased a subject's vault values` with the command ID, and keeps a proof in the vault's `erasures` table: the SHA-256 of the subject, the erasing command, how many values went, and when. It does not keep the subject itself: ```sh sqlite3 vault.db "SELECT subject_hash, tenant, command_id, erased, datetime(at / 1000, 'unixepoch') FROM erasures" printf '%s' c1 | shasum -a 256 # the subject_hash of c1 sqlite3 vault.db "SELECT count(*) FROM vault WHERE subject = 'c1'" # 0 ``` Then check that nothing personal is stored in clear. Payloads in the SQLite store are JSON text, so you can search them for a test person's email; it must not appear: ```sh sqlite3 kernable.db "SELECT count(*) FROM events WHERE payload LIKE '%ann@example.com%'" # 0 sqlite3 kernable.db "SELECT count(*) FROM commands WHERE payload LIKE '%ann@example.com%'" # 0 ``` The vault values are encrypted, so a search of `vault.db` finds nothing even before an erasure. The kernel UI shows the references, never the values. ## What stays The events and commands stay, with references instead of values, and the hash chain still verifies: you keep the history (an order was placed and confirmed) without the person's data. These are not erased: - **IDs**: aggregate IDs, command IDs, and the subject that the erasing event names. Use IDs that do not identify a person by themselves, not email addresses. - **The request context** of commands, such as `senderIdentity`, stored in clear with every command. - **Fields not marked personal**, and **files and cache values**. --- # Passwords and other secrets Use this page when a command carries a value that must never be stored: a password, a one-time code, a token. Mark the field **secret**: the handler sees it in clear, the log never does, and the kernel deletes it as soon as the command is done. ## Set it up 1. **Turn the vault on:** start the kernel with `--vault-key` (`kernable start --dev` does). Secrets are kept in the vault while their command runs; without a vault, the kernel refuses commands with secret fields (`FAILED_PRECONDITION`). 2. **Mark the field secret** on the command, as the shop does for `SetPassword` and `SignIn` (highlighted below). Only top-level fields of commands can be secret. 3. **Give the client the domain**, so it seals the field before signing: `new Kernable().use(shopDomain)`, Python `Kernable().use(shop_domain)`, Rust `.with_domain(&shop_domain)`. A client without it sends the password in clear, and it is stored in the command log for good. `kernable submit` cannot seal fields: never use it for such commands. 4. **Store a hash, not the value**, in the handler (see the rules below). The example stores an scrypt hash in `PasswordSet`, marked personal, so it is erased with the customer. domain.ts: ```7 // Personal data goes to the kernel's vault; the signed log keeps references only. shopDomain.command("Register").personal(["email", "name"], (_d, id) => id); shopDomain.event("CustomerRegistered", 1).personal(["email", "name"], (_d, id) => id); shopDomain.command("Confirm").personal(["email"], (d: { customerId: string }) => d.customerId); shopDomain.event("OrderConfirmed", 1).personal(["email"], (d: { customerId: string }) => d.customerId); // Passwords are secrets: the handler sees them once; the vault deletes them when it is done. shopDomain.command("SetPassword").secret(["password"]); shopDomain.command("SignIn").secret(["password"]); shopDomain.event("PasswordSet", 1).personal(["hash", "salt"], (_d, id) => id); // Erasing a customer deletes their personal data everywhere, in every stream. shopDomain.event("CustomerErased", 1).erases((_d, id) => id); const hash = (password: string, salt: string) => scryptSync(password, salt, 32).toString("hex"); customerAggregate .apply("PasswordSet", (s, data: { hash: string; salt: string }) => { s.passwordHash = data.hash; s.salt = data.salt; }) .apply("CustomerErased", (s) => { delete s.passwordHash; delete s.salt; s.registered = false; }) .command("SetPassword", (s, data: { password: string }, _cmd, ctx) => { if (!s.registered) throw new Rejected("No such customer", "not_found"); if ((data.password ?? "").length < 8) throw new Rejected("At least 8 characters", "invalid"); const salt = ctx.id(); // deterministic: the command may run again return [{ type: "PasswordSet", data: { hash: hash(data.password, salt), salt } }]; }) .command("SignIn", (s, data: { password: string }) => { if (!s.passwordHash || hash(data.password ?? "", s.salt!) !== s.passwordHash) throw new Rejected("Wrong customer or password", "denied"); return [{ type: "SignedIn" }]; }) .command("Erase", (s) => { if (!s.registered) throw new Rejected("No such customer", "not_found"); return [{ type: "CustomerErased" }]; }); ``` domain.py: ```7 # Personal data goes to the kernel's vault; the signed log keeps references only. shop_domain.command("Register").personal(["email", "name"], lambda d, customer_id: customer_id) shop_domain.event("CustomerRegistered", 1).personal(["email", "name"], lambda d, customer_id: customer_id) shop_domain.command("Confirm").personal(["email"], lambda d: d["customerId"]) shop_domain.event("OrderConfirmed", 1).personal(["email"], lambda d: d["customerId"]) # Passwords are secrets: the handler sees them once; the vault deletes them when it is done. shop_domain.command("SetPassword").secret(["password"]) shop_domain.command("SignIn").secret(["password"]) shop_domain.event("PasswordSet", 1).personal(["hash", "salt"], lambda d, customer_id: customer_id) # Erasing a customer deletes their personal data everywhere, in every stream. shop_domain.event("CustomerErased", 1).erases(lambda d, customer_id: customer_id) def password_hash(password: str, salt: str) -> str: return hashlib.scrypt(password.encode(), salt=salt.encode(), n=2**14, r=8, p=1, dklen=32).hex() @customer_aggregate.apply("PasswordSet") def _(state, data): state["password_hash"] = data["hash"] state["salt"] = data["salt"] @customer_aggregate.apply("CustomerErased") def _(state, data): state.pop("password_hash", None) state.pop("salt", None) state["registered"] = False @customer_aggregate.command("SetPassword") def _(state, data, cmd, ctx): if not state["registered"]: raise Rejected("No such customer", "not_found") if len(data.get("password") or "") < 8: raise Rejected("At least 8 characters", "invalid") salt = ctx.id() # deterministic: the command may run again return [{"type": "PasswordSet", "data": {"hash": password_hash(data["password"], salt), "salt": salt}}] @customer_aggregate.command("SignIn") def _(state, data): if "password_hash" not in state or password_hash(data.get("password") or "", state["salt"]) != state["password_hash"]: raise Rejected("Wrong customer or password", "denied") return [{"type": "SignedIn"}] @customer_aggregate.command("Erase") def _(state, data): if not state["registered"]: raise Rejected("No such customer", "not_found") return [{"type": "CustomerErased"}] ``` domain.rs: ```29 #[derive(Deserialize)] struct Password { password: Option, } #[derive(Serialize, Deserialize)] struct PasswordSet { hash: String, salt: String, } fn vault(shop_domain: &Domain, customer_aggregate: &Aggregate) { let aggregate = |_: &Value, id: &str| id.to_owned(); let customer_id = |d: &Value, _: &str| d["customerId"].as_str().unwrap_or_default().to_owned(); // Personal data goes to the kernel's vault; the signed log keeps references only. shop_domain .command("Register") .personal(&["email", "name"], aggregate); shop_domain .event("CustomerRegistered", 1) .personal(&["email", "name"], aggregate); shop_domain .command("Confirm") .personal(&["email"], customer_id); shop_domain .event("OrderConfirmed", 1) .personal(&["email"], customer_id); // Passwords are secrets: the handler sees them once; the vault deletes them when it is done. shop_domain.command("SetPassword").secret(&["password"]); shop_domain.command("SignIn").secret(&["password"]); shop_domain .event("PasswordSet", 1) .personal(&["hash", "salt"], aggregate); // Erasing a customer deletes their personal data everywhere, in every stream. shop_domain.event("CustomerErased", 1).erases(aggregate); customer_aggregate.apply("PasswordSet", |s, data: PasswordSet, _evt, _ctx| { s.password_hash = Some(data.hash); s.salt = Some(data.salt); }); customer_aggregate.apply("CustomerErased", |s, _: Value, _evt, _ctx| { *s = Customer::default() }); customer_aggregate.command("SetPassword", |s, data: Password, _cmd, ctx| { if !s.registered { return Err(Rejected::new("No such customer").code("not_found")); } let password = data.password.unwrap_or_default(); if password.chars().count() < 8 { return Err(Rejected::new("At least 8 characters").code("invalid")); } let salt = ctx.id(); // deterministic: the command may run again Ok(vec![NewEvent::new( "PasswordSet", PasswordSet { hash: hash(&password, &salt), salt, }, )]) }); customer_aggregate.command("SignIn", |s, data: Password, _cmd, _ctx| { let password = data.password.unwrap_or_default(); match (&s.password_hash, &s.salt) { (Some(stored), Some(salt)) if hash(&password, salt) == *stored => { Ok(vec![NewEvent::new("SignedIn", ())]) } _ => Err(Rejected::new("Wrong customer or password").code("denied")), } }); customer_aggregate.command("Erase", |s, _: Value, _cmd, _ctx| { if !s.registered { return Err(Rejected::new("No such customer").code("not_found")); } Ok(vec![NewEvent::new("CustomerErased", ())]) }); } /// scrypt with the parameters of the TypeScript and Python shops, so their hashes match. fn hash(password: &str, salt: &str) -> String { let params = scrypt::Params::new(14, 8, 1, 32).expect("valid scrypt parameters"); let mut out = [0u8; 32]; scrypt::scrypt(password.as_bytes(), salt.as_bytes(), ¶ms, &mut out).expect("32 bytes fit"); out.iter().map(|b| format!("{b:02x}")).collect() } ``` ## What happens to the value - The client moves the field to the vault before signing the command; the stored command keeps a reference, `{"$vault": "…"}`. The reference is part of the command's signature: nobody can swap the value without breaking it. - The worker gets the value in clear for as long as the command runs, also when it is delivered again after a crash. So do middlewares granted `vault_read` for the command. - The vault deletes it as soon as the command completed, committed or rejected. A kernel that restarts deletes those of commands that completed meanwhile. A command that waits in the queue keeps its secret until it completes. - A middleware that rejects the command before it is queued leaves nothing behind: the value never reaches the vault. Check it in the stores: the command holds the reference only, and no secret is left once no command runs. ```sh sqlite3 kernable.db "SELECT payload FROM commands WHERE command_type = 'SetPassword'" # {"password":{"$vault":"…"}} sqlite3 vault.db "SELECT count(*) FROM vault WHERE secret = 1" # 0 ``` ## Rules for handlers - **Never put a secret into an event**, a log line or an error message. Store a hash, salted and slow (scrypt, Argon2), and mark it personal so it is erased with the person ([Handle an erasure request](./erasure)). - **Derive the salt with `ctx.id()`.** Handlers must decide the same way when a command runs again; a random salt would differ. - **Compare in a command**, like `SignIn`: the stored hash stays in the aggregate, and the answer is the outcome (committed, or rejected with `denied`). Use the same message for an unknown person and a wrong password. - **One-time codes:** store a hash with an expiry in an event, and send the code itself as a secret in the command that uses it. How the vault works: [Personal data: the vault](../concepts/data#personal-data-the-vault). --- # Store files and cache values Use this page when a process needs to keep something that is not an event: a **file** of a domain (an invoice PDF, an upload) or a **cache value** that may expire (a rate limit, a computed answer). The kernel keeps both for your processes, so you need no separate file server or cache. Both are kept under tenant, domain and key, and every call is signed and checked against the key's grant, like a query. Neither is part of the signed event log: they are not hash-chained, not verified by `kernable verify`, and **not erased with the vault**. If a file holds personal data, delete it yourself when you erase the person ([below](#personal-data-in-files)). | | Files | Cache | |---|---|---| | Kept | until deleted | until they expire, are deleted, or the kernel restarts | | Largest value | 16 MiB (`--max-data-bytes`, up to 128 MiB) | 1 MiB | | Key | 1 to 512 bytes | 1 to 512 bytes | | Grant | `data_read`, `data_write` | `cache` | | Flag | `--data-store` (default `file:data`) | `--cache-store` (only `memory:`) | ## 1. Grant access | Action | Allows | |---|---| | `data_read` | getting and listing the domain's files | | `data_write` | putting and deleting them | | `cache` | setting, getting and deleting cache values | Grant them per domain: ```sh kernable grant keys/root.key keys/api.pub 'data_write=shop.*' 'data_read=shop.*' 'cache=shop.*' ``` `kernable start --dev` lets `app` do everything. A key without the grant gets `PERMISSION_DENIED` (`key may not read files in shop`). More on grants: [Keys per service](./keys). Grants decide per key and domain; **who** may read which file, and in which tenant, a middleware decides. Every file and cache request carries the client's request context (its default one, or one per call) and passes the middleware chain under the types `shop.PutData`, `shop.GetData`, `shop.ListData`, `shop.DeleteData`, `shop.SetCache`, `shop.GetCache` and `shop.DeleteCache`. A middleware covering `shop.*` checks them like commands; a refusal is `PERMISSION_DENIED` with its code. Use it, for example, to keep tenants apart: refuse when `senderTenant` is not the request's tenant ([Queries and middleware](../concepts/queries-and-middleware#middleware)). ## 2. Put and get files Any client can use them: `kernable.data` on a `Kernable` (`new Kernable()`, Python `Kernable()`, Rust `Kernable::new(…)`). ```ts [TypeScript] const sha256 = await kernable.data.put("shop", "invoices/o-1.pdf", pdf, { contentType: "application/pdf" }); const file = await kernable.data.get("shop", "invoices/o-1.pdf"); // { data, contentType, sha256 } or undefined const { files, next } = await kernable.data.list("shop", "invoices/"); // [{ key, contentType, size }] await kernable.data.delete("shop", "invoices/o-1.pdf"); // whether it existed ``` ```python [Python] sha256 = k.data.put("shop", "invoices/o-1.pdf", pdf, "application/pdf") file = k.data.get("shop", "invoices/o-1.pdf") # Data(data, content_type, sha256) or None page = k.data.list("shop", "invoices/") # Listing(files=[Listed(key, content_type, size)], next) k.data.delete("shop", "invoices/o-1.pdf") # whether it existed ``` ```rust [Rust] let sha256 = kernable.data().put("shop", "invoices/o-1.pdf", pdf, "application/pdf").await?; let file = kernable.data().get("shop", "invoices/o-1.pdf").await?; // Option let page = kernable.data().list("shop", "invoices/", "").await?; // Listing { files, next } kernable.data().delete("shop", "invoices/o-1.pdf").await?; // whether it existed ``` - Putting a file replaces the one under the same key. A reader gets the old file or the new one, never half of it. - Text is stored as UTF-8. Without a content type, a string is stored as `text/plain; charset=utf-8`, bytes as `application/octet-stream`. - The tenant is the client's; TypeScript (`{ tenant }`) and Python (`tenant=`) can name another per call. - A put and a get answer the SHA-256 of the file's data (hex), as the kernel computed it. Name it in the event that records the file (`InvoiceStored { key, sha256 }`): the signed log then says which bytes belong to it, and a reader can check them. - A listing gives the files whose keys start with a prefix (empty for all of the domain), with their content type and size, in the order of their keys' bytes, up to 1,000 per page; pass `next` as `after` for the next page. Use it to find files no event names (a crash between a put and the command that records it leaves one), or to see what a tenant stores. Keys you can compute again, such as `invoices/.pdf`, still need no listing to be found. - Each file travels whole in one message: the kernel and the SDK hold it in memory while it is put or read. Raise `--max-data-bytes` (up to 128 MiB) for larger files; for anything bigger, keep the bytes in object storage and their address and hash in an event. - A put or delete runs at most once per signed request. One that failed may still have taken effect, so the same request is refused as a replay (`ALREADY_EXISTS`); send it again as a new call, which the SDKs sign anew. ## 3. Set and get cache values ```ts [TypeScript] await kernable.cache.set("shop", "quote/c-1", JSON.stringify(quote), { ttlMs: 60_000 }); const cached = await kernable.cache.get("shop", "quote/c-1"); // bytes or undefined ``` ```python [Python] k.cache.set("shop", "quote/c-1", json.dumps(quote), ttl_ms=60_000) cached = k.cache.get("shop", "quote/c-1") # bytes or None ``` ```rust [Rust] kernable.cache().set("shop", "quote/c-1", serde_json::to_vec("e)?, Some(Duration::from_secs(60))).await?; let cached = kernable.cache().get("shop", "quote/c-1").await?; // Option> ``` Without a TTL (`0`, `None`), a value lives until it is deleted or the kernel restarts. A restart always empties the cache: keep in it only what you can compute again. Every process connected to the kernel sees the same values. ## Personal data in files An erasure deletes vault values only. To delete a person's files with them, handle the erasing event in a reactor whose key may `listen` to it and has `data_write`, and delete the files under keys you can derive from the person: ```ts const erasedFiles = shopDomain.reactor("erased-files") .on("CustomerErased", async (_data, evt) => { await kernable.data.delete("shop", `customers/${evt.aggregateId}/avatar.png`); }); await kernable.listener(erasedFiles).run(); ``` Deleting a missing file is no error, so a redelivered event does no harm. Keep personal data out of cache values, or give them a short TTL. See [Handle an erasure request](./erasure). ## Where they live | Flag | Values | Notes | |---|---|---| | `--data-store` | `file:` (default `file:data`), `memory:` | The directory is created on the first file. Each file is `///`, every part hex-encoded; a part over 123 bytes, whose hex would not fit in a file name, is `h-` and the SHA-256 of it instead. Each file starts with its tenant, domain, key and content type, so a listing and a standby know every name. `memory:` loses the files on a restart. | | `--cache-store` | `memory:` | The only backend. | A listing reads the domain's directory, so it takes longer the more files a tenant keeps in a domain. Back up the data directory together with the event store ([Back up and restore](./backup)). All flags: [Configuration](../reference/configuration). ## Watching them The UI's **Explore → Data** shows the files per tenant and domain (how many, their size, and a domain as folders by `/` in its keys) and the cache (entries, memory against `--max-cache-bytes`, the hit rate, values that expired or were evicted, and the keys with how long until they expire). It lists names and sizes only: it never opens a file or shows a cache value ([Monitor kernable](./monitor#the-ui)). --- # Write an SDK for another language Any language with gRPC and Ed25519 can take part. The TypeScript SDK is the reference implementation: `sdks/typescript/src/rpc.ts` (connection), `signing.ts` (byte layouts) and `index.ts` (everything else). Work through the steps in order; each one builds on the one before. The details live in the reference: [Protocol](../reference/protocol), [Signatures](../reference/signing), [Errors](../reference/errors), and the [TypeScript](../reference/sdk-typescript), [Python](../reference/sdk-python) and [Rust](../reference/sdk-rust) SDKs for names and options. ## 1. Signatures first Implement the byte layouts in [Signatures](../reference/signing) before anything else; nothing else works without them. - Keys: a `.key` file holds a 32-byte Ed25519 seed in hex. The **key ID** is the lowercase hex public key. - Messages: a `str` tag, then length-prefixed strings and bytes and big-endian integers. Payloads are signed as the exact bytes sent, with no JSON canonicalization. - Also implement **derived IDs** (`kernable.v1.id`, for `ctx.id()`) and **vault references** (`kernable.v1.vault`). Test against `proto/test-vectors/signing.json` (see `sdks/typescript/test/signing.test.ts`). Every message and signature must come out byte for byte: `command`, `command_vault`, `query`, `result`, `result_vault`, `rejection`, `snapshot`, `challenge`, `subscribe`, `data_put`, `cache_set`, `membership`, `checkpoint`, `revocation`, `event_hash`, `event_hash_vault`, `derived_id` and `vault_ref`. Skip `grant` and `revoke`: only root-key tooling makes them. ## 2. Connect Generate gRPC stubs from `proto/kernable/v1/kernable.proto`, package `kernable.v1`. | Setting | Option | Else | Else | |---|---|---|---| | Address | `url` | `KERNABLE_URL` | `http://127.0.0.1:7233` | | Key file | `key` | `KERNABLE_KEY` | `app.key` in the current directory | | Grant file | `grant` | `KERNABLE_GRANT` | the `.grant` file next to the key (`app.key` → `app.grant`), if it exists | | Mutual TLS | `tls` | `KERNABLE_TLS_CA`, `KERNABLE_TLS_CERT`, `KERNABLE_TLS_KEY` (all three or none) | plain connection, on this machine only | | Plain text off the machine | `insecure` | `KERNABLE_INSECURE=1` | refused | | Request timeout | `timeoutMs` (or your language's unit) | — | 60 s | | Service | `service` | `KERNABLE_SERVICE` | none | - Accept `http://host:port`, `host:port` and, on Unix, `unix:/path`. - **Grant:** send the grant file's content (trimmed) as the gRPC metadata entry `kernable-grant` on every call and every stream opening. It is a hex-encoded `Grant`; send it as it is. Without a grant file, send no entry: the kernel then uses the grant it already holds for the key, if any. - **Protocol version:** set `protocol: 1` on every request and stream opening (`SubmitRequest`, `QueryRequest`, `SubscribeRequest`, `Register`, `Hello`, and the data, cache and apply requests). Export it as a constant (`PROTOCOL`). The kernel refuses other versions with `FAILED_PRECONDITION`. - Raise the message size limit for `DataService`: a file can be 16 MiB. - **Plain text stays on the machine:** without TLS, refuse a URL that is not a Unix socket or a loopback address (`localhost`, `127.x.x.x`, `::1`) unless `insecure` is set. - **Requests wait for the connection:** every unary call (submit, query, data, cache, apply) waits while the kernel cannot be reached (gRPC's wait-for-ready) and has a deadline, 60 s by default, above the kernel's own submit and settle timeouts; submit and query take their own. Never retry a call that may have reached the kernel: a data or cache write sent twice is refused as a replay. Give every request an `issued_at` later than the one before it in the process (now, or the last one plus 1 ms): two calls in the same millisecond would otherwise carry the same signature, and the second would be refused as a replay. - **Keepalive:** ping every 30 s, also when idle, and drop the connection after 10 s without an answer, so streams on a connection that died silently reconnect. ## 3. Clients All of these are unary calls, except `Subscribe`. - **`Submit`** (`CommandService`). Build the `Command` with a new UUID as `command_id` unless the caller gives one (it is the idempotency key), default the context's `correlation_id` to the command ID, seal the vault fields (step 7), sign `kernable.v1.command`, and send it with a wait mode: - `WAIT_SETTLED` (the default): return the events; if `settled` is false, raise a **NotSettled** error that carries them. - `WAIT_COMMITTED`: return the events. - `WAIT_ACCEPTED` (`enqueue`): return the command ID. - A `rejected` outcome is a normal result (`committed: false`, `code`, `message`), not an error. - **`Query`** (`QueryService`). Set `issued_at` to now, sign `kernable.v1.query`, send. Return the decoded JSON `response`; raise **Rejected** (with `code` and `message`) on a rejection. - **`Subscribe`** (`EventService`). Sign `kernable.v1.subscribe` with `after_position`, the event types and `issued_at`. Hand events to the handler one at a time. On any stream end or error, reconnect with `after_position` set to the last handled event, signed again with a new `issued_at`. - **Data and cache** (`DataService`, `CacheService`). Sign `kernable.v1.data.` or `kernable.v1.cache.` per call; get and delete sign the hash of empty data. Payloads are JSON, encoded as UTF-8 bytes; an empty payload reads as `{}`. ## 4. Workers Open `WorkerGateway.Session` and send `Register`: - `aggregates`: each aggregate as a `WorkerAggregate`, with its qualified type (`shop.Cart`) and qualified command types (`shop.AddItem`). - `event_schemas`: the payload version this worker writes for every event type its aggregates know (step 8). - `service`, `key_id`, `worker_id` (for logs), `max_concurrency` (TypeScript: 16), `protocol`. Answer the `Challenge` with a signature over `kernable.v1.challenge` and the nonce; `Registered` means the worker is connected. Then, for every `Execute`, in parallel up to `max_concurrency`: 1. Find the aggregate by command type; else reject with `unknown_command`. 2. If there is a `snapshot` whose `schema` is not the aggregate's, reject with `snapshot_stale`: the kernel runs the command again with every event. 3. Fold the snapshot state (or the initial state) and the `history`, upcast and unsealed with the `vault` values. 4. Run the handler with the unsealed command. `now()` is `received_at`; `id()` is the n-th derived ID of the command. 5. Seal the new events' vault fields, sign `kernable.v1.result` and send a `Result` with the events, their vault values and, optionally, a signed snapshot. 6. If the handler fails, sign `kernable.v1.rejection` and send a `Rejection`: the handler's own `Rejected` code and message, else `handler_error` with only the error's type as the message. The rejection is stored for good and returned to the sender, and an error's text can hold personal data; log the whole error instead. Give the handler a copy of the state: the snapshot is computed from the state after the new events, and a handler that changed the state in place would sign a snapshot no replay produces. Catch everything that can go wrong outside the handler too (a malformed `Execute`, an eraser that fails), so a command is always answered or the session ends. If the worker meets an event with a newer schema than it knows, it must not reject the command: log an error and end the session. The worker needs an update. ## 5. Listeners Open `ListenerService.Listen` and send `Hello`: `name`, `key_id`, `order`, qualified `event_types` and `query_types`, `middlewares` (at most one per `when`: `order`, types, `WHEN_BEFORE` or `WHEN_AFTER`), `service`, `live`, `protocol`, `instance` and `single` ([instances](./instances): take `instance` from `KERNABLE_INSTANCE` unless the listener names one; `single` defaults to true for reactors, false for read models, and is never set without an instance or for a live listener), and `from_position` only when you want to start somewhere else than the stored position (on a reconnect, or for a read model with its own store). Answer the challenge, then wait for `Ready`: events up to `head_position` are a replay. An instance of a single group may get `Waiting` instead while another instance is active: count it as started (it must not hold up the process), log the active instance, and keep the stream open; `Ready` follows when it takes over. Resume from the last handled position only after a `Ready`: an instance that only waited has none, and must not send `from_position` 0 for a reactor. Handle the deliveries **one at a time, in order**: | Delivery | What to do | |---|---| | `Batch` | Run the handlers for its `retries` (parked events an operator asked for again, regardless of the position), then for its events (upcast, unsealed with the batch's `vault`), then send `Ack` with `up_to` and the events it parked. With a store, commit the data and `up_to` together before the ack. | | `QueryCall` | Run the query handler and send a `QueryAnswer` with the JSON response, or a rejection: the handler's code, `unknown_query` if there is no handler, `handler_error` if it fails. | | `MiddlewareCall` | Run the middleware for its `when` and send a `Verdict`: empty to pass, else a rejection (`not_a_middleware` if there is none for that `when`, `middleware_error` if it fails). After calls carry the outcome. | **Delivery is at least once.** If a handler fails, end the stream and reconnect with `from_position` set to the last handled position: the kernel delivers the rest again. Handlers must be safe to run twice. Batches may arrive before the previous one is acknowledged: once one fails, drop the ones queued behind it, and end the connection only after the one being handled, so two batches never run at once. **Parking.** A listener may give up on an event and move on: list it in the `Ack` as a `Parked` with its position, the code and message of a `Rejected` (for another error `handler_error` and only the error's type: the kernel keeps it, and an error's text can hold personal data), and how often it was handled. Park a `Rejected` at once and other errors after a number of attempts (the SDKs: 5 for reactors, never for read models, an option per listener); never park while the kernel is unreachable or busy (`UNAVAILABLE`, `DEADLINE_EXCEEDED`, `RESOURCE_EXHAUSTED`) or the store fails, and never in a live listener or for an event that erases a subject (its handler must get past it, or what it removes stays behind). With a store, drop the parked event's writes and keep the rest of the batch (a savepoint per event). A retried event that fails again is parked again; one that is handled leaves the kernel's list with the ack. Give handlers `ctx.submit` (requires a fixed ID, follows the event, raises a rejection) and `ctx.query` (the event's tenant, the client's context). **Never go back.** Skip events at or below the last handled position, and end the session, without an ack, on a batch whose `up_to` is below it: a buggy kernel, or something posing as one, must not make a store apply events twice. Send `from_position` only once a session reached `Ready` (or from a store that committed something): a new reactor whose first connect fails must still start at the newest event, not at 0. **With a store**, empty SQLite's write-ahead log when you open it (a crash after an erasing batch may have left the removed values' old pages there), and write the batch's data and position in one transaction; if the commit fails, roll back and resume from the stored position, and if a batch was abandoned half-way, roll back what is left of it before the next one. Submits and queries sent from inside a handler must set `listener` to the listener's name, so the kernel never makes them wait for the listener that is busy sending them. A context for a follow-up command keeps the event's correlation ID and sets `causation_id` to the event's command ID. ## 6. Reconnect **Partitions.** Ask the first kernel for its map (`PartitionService.Map`; `NOT_FOUND`: it serves every tenant). Send each submit, query, file and cache request to its tenant's partition (listed in `tenants`, else `default`), memberships to `control`. On `FAILED_PRECONDITION` with a message starting `wrong_partition`, read the map again and send the request anew. Run workers, listeners and subscriptions on every partition; a listener's store keeps a position per partition, and its batches take turns on it. For a query option `allPartitions`, first send the query with `wait_only` set (signed, an empty payload, `min_position` 0) to every partition at once, then the query itself to the tenant's partition; each wait-only query returns an empty response once a read model that answers the type has processed that partition's head. See [Share tenants across kernels](./partitions). **Several addresses.** Accept a comma-separated list of kernel addresses: the two kernels of a pair ([Run a standby](./standby)). Keep one current address for all calls. When a call or stream on it ends with `UNAVAILABLE` (a standby refuses everything with `not_primary`) or cannot connect, make the next address current. A unary request tries the next address within its timeout, waiting at most about a second for each to connect; a stream reconnects to the next one. Do not parse the address in `not_primary`: the kernels may know each other by other names than your clients. Workers, listeners and subscriptions run until stopped. When a stream ends or fails for any reason, log it and reconnect after a delay (TypeScript: 1 s, option `reconnectMs`), twice as long after each further failed attempt, up to 30 s, give or take a fifth so processes do not reconnect in lockstep; start over once a session is accepted. Log refusals a retry cannot change (`INVALID_ARGUMENT`, `PERMISSION_DENIED`, `FAILED_PRECONDITION`, `UNIMPLEMENTED`, `UNAUTHENTICATED`) as errors. A listener resumes from its last handled position, a subscription from its last handled event. Commands in flight when a worker drops go to another worker once their lease expires. ## 7. Vault fields A domain declares **personal** fields (with a subject, from the payload and the aggregate ID) and **secret** fields per command and event type. - **Seal** before signing: replace each set field with `{"$vault": ref}`, where `ref` is the vault reference over 16 random salt bytes, the subject (empty for secrets), `secret`, and the field's JSON. Send the values next to the command (`SubmitRequest.vault`) or result (`Result.vault`) and list the refs in `vault_refs`, in payload order. - **Unseal** on reading (`Execute`, `Batch`, `MiddlewareCall`): put the values back; a reference without a value reads as `null`. - An event type that erases a subject sets `erases` on its `NewEvent`. ## 8. Event versions A domain declares per event type its current `schema` and an upcast from each older version. Write events with their schema; before handing an event to a fold or a handler, upcast it step by step. An event whose schema is newer than known, or with a missing step, is an error, not something to skip. See [Change an event's format](./event-versions). ## 9. Memberships and apps - **`apply(manifest)`** sends a `Membership` to `MembershipService.Apply`: the service, the qualified aggregate types of its workers, the names of its listeners and of its middlewares, metadata in the given order, and `issued_at` now, signed over `kernable.v1.membership`. The key needs `manage` for the service. - **`app({ service, version, description, domains })`** runs whole domains as one service. Its `run()` applies the membership (all aggregates as workers; durable listeners without middlewares as listeners, those with one as middlewares; live listeners not listed; `version` and `description` in the metadata), then runs one worker and every listener. See [Remove a listener, middleware or aggregate](./remove-listener). ## 10. Errors Pass the kernel's gRPC status to the caller with its code, so callers can act on it ([Errors](../reference/errors)): | What | Where | SDK | |---|---|---| | `NOT_FOUND` | submit | No worker registered the aggregate or command type, and no membership lists it. Raise it. | | `UNAVAILABLE` | submit, query | A middleware is away, no read model answers, or the kernel is shutting down. Raise it; the caller retries. On a stream: reconnect. | | `PERMISSION_DENIED` | any call or stream | The grant does not allow it, or the key is revoked. Raise it; on a stream, log and reconnect. | | `FAILED_PRECONDITION` | any | Unsupported protocol version, an outdated worker, or a handler request that would wait for its own listener. Raise it. | | `version_mismatch` and other rejections | submit outcome | Not an error: return `committed: false` with the code. | | `handler_error` | worker, read model | What the SDK sends when a handler fails with something other than `Rejected`; for a command, with the error's type only. | ## 11. Logs Print connection and activity lines to stderr as `HH:MM:SS what line`, for example `12:04:31 listener carts connected to …`, with the level from `KERNABLE_LOG` (`off`, `error`, `warn`, `info` by default, `debug`) and colors only on a terminal without `NO_COLOR`. An app prints one overview instead, once everything is connected; match the format in [Getting started](../tutorial/getting-started). ## 12. Prove it works 1. The signing test vectors pass (step 1). 2. Port the tutorial's shop (`docs/examples/shop/ts`) to your language, with the same output as the `.out` files. 3. Run it in [part 5's](../tutorial/shop/5-languages) mix: your worker with the TypeScript read models and the Rust reactor, and every other combination (see `docs/examples/shop.test.ts`). If it works next to the TypeScript, Python and Rust processes, it speaks kernable. --- # The kit Almost every application needs the same groundwork before its own domain: tenants for its customers, people who sign in, their roles, invitations, and a check on every request. The kit is that groundwork as source code: `kernable-sdk add kit` copies it into your project, and from then on it is yours. Change fields, rules and flows, or delete what you do not need. It is written for TypeScript, Python and Rust. All three write the same events and pass the same test cases, so an API in TypeScript and a service in Rust work on one log. ## What is in it | Domain | What it holds | Page | |---|---|---| | `tenancy` | One tenant per customer organisation: its slug, status, owners, MFA requirement, and the platform's sign-up policy | [Tenancy](./tenancy) | | `account` | A person's account, the same in every tenant: email, password, TOTP, passkeys, OAuth/OIDC logins; a session per sign-in; a tenant's single sign-on | [Accounts and sessions](./account) | | `identity` | A person or service inside one tenant: profile, group memberships, API keys | [Identities](./identity) | | `group` | Named sets of permissions per tenant | [Groups](./group) | | `invitation` | Invitations by email into a tenant, with groups or as owner; invitation links many people join through | [Invitations](./invitation) | | authorization | The middleware that decides every request, relations, lists filtered by the same rules, `Explain` | [Authorization](./authorization) | | `KitApi` | What only your HTTP API does: sign-in, tokens, passkey challenges, OIDC, API keys | [Your API: KitApi](./api) | ## How the parts fit ```mermaid flowchart LR subgraph platform["platform tenant"] A["account
(email, password, passkeys)"] S["session
(one per sign-in)"] end subgraph t["a customer's tenant"] T["tenant
(slug, owners, MFA)"] I["identity
(profile, memberships)"] G["group
(permissions)"] V["invitation"] end A -- "signs in" --> S A -- "one per tenant" --> I I -- "member of" --> G T -- "owners" --> I V -- "accepted: creates" --> I M["authorization
middleware"] -. "decides every request
from these events" .-> I ``` - An **account** is the person: one email, one password, their second factors. It lives in the platform tenant, so the same person signs in once and works in several tenants. - An **identity** is that person *inside one tenant*. Permissions belong to identities, not accounts, so the same person can own one tenant and be a guest in another. Services (other programs) have identities too, with API keys instead of an account. - A **group** is a named set of permissions in a tenant, such as "Customers may place orders and cancel their own". A tenant's **owners** hold every permission in it; they form a virtual group. - The **middleware** runs before every command, query, file and cache request. It learns from the kit's events and decides from what it knows, without a database call. ## What "yours" means The kit is not a dependency you update. `add kit` copies it once, with the version of your SDK; each copied file says which version it came from. Every change it makes is an event in your log, so the kit's audit trail is your log: sign-ins, failed attempts, memberships, invitations. A newer kit is never installed over yours. To take something from it, copy it into another directory and move over what you want. The kit's test cases (`kit/fixtures`, the same JSON for all three languages) tell you which of its rules you changed. ## Start here 1. [Getting started](./getting-started): copy the kit, run it, grant its keys, sign people in. 2. The page of each domain, for its rules, flows and settings, with a generated reference of its commands, events and queries at the end. 3. [Recipes](./recipes) for common changes: sign-up modes, enforced MFA, single sign-on, API keys for services, removing a domain you do not need, adding a field. Personal data (emails, names, IP addresses, password hashes) lives in the [vault](../concepts/data) and is erased with the person; the kit's events mark which fields go there. Read [Trust](../concepts/trust) first if keys, grants and the request context are new to you. --- # Getting started with the kit This page takes you from an empty directory to people signing in to your API: copy the kit, run its domains next to yours, grant its keys, send its mail, and turn a signed-in person into the request context of your own commands. [The kit](./) explains what each domain holds. ## 1. Copy it ```sh [TypeScript] npx @kernable/sdk add kit # into ./kit; or: add kit npm install @node-rs/argon2 @simplewebauthn/server openid-client node --test kit/test/unit/*.test.ts ``` ```sh [Python] kernable-sdk add kit # into ./kit, a package; or: add kit pip install -r kit/requirements.txt python -m unittest discover -s kit/tests -t . ``` ```sh [Rust] cargo install kernable-sdk kernable-sdk add kit # the crate ./kit; or: add kit cargo test -p kit # after adding it to your workspace ``` The kit comes with the SDK, so its version matches your SDK's. The tests are the kit's contract: the cases every language's kit passes, as JSON in `kit/fixtures`. They pass in the kit as shipped; when you change a rule, change its case too. ## 2. Run it The kit is a set of [domains](../concepts/domains). Declare the relations your permissions use, then run the kit's domains next to yours: ```ts [TypeScript] import { Kernable } from "@kernable/sdk"; import { kitDomains, relation, useAuthorizationStore } from "./kit/src/kit.ts"; import { shopDomain } from "./shop/domain.ts"; // Who stands in which relation to your aggregates, from your events. relation("shop.order", "owner").set("shop.OrderPlaced", (e) => e.data.customer); useAuthorizationStore("sqlite:authorization.db"); // the middleware resumes after a restart const app = new Kernable().app({ service: "kit", domains: [...kitDomains(), shopDomain] }); await app.run(); ``` ```python [Python] from kernable import Kernable from kit import kit_domains, relation, use_authorization_store from shop.domain import shop_domain # Who stands in which relation to your aggregates, from your events. relation("shop.order", "owner").set("shop.OrderPlaced", lambda e: e.data["customer"]) use_authorization_store("sqlite:authorization.db") # the middleware resumes after a restart Kernable().app(service="kit", domains=[*kit_domains(), shop_domain]).run() ``` ```rust [Rust] use kernable::{App, Kernable}; use kit::authorization::{ids, relation, use_authorization_store}; // Who stands in which relation to your aggregates, from your events. relation("shop.order", "owner").set("shop.OrderPlaced", |e| ids(&e.data["customer"])); use_authorization_store("sqlite:authorization.db"); // the middleware resumes after a restart let mut app = App::new().service("kit"); for d in kit::kit_domains().iter().chain([&shop_domain]) { app = app.domain(d); } Kernable::new("kit.key")?.app(app)?.run().await?; ``` Once, `bootstrap()` creates the platform tenant (where accounts and the platform's administrators live), the sign-up policy, the identity and key your API uses for itself, and optionally the first administrator. It may run again; it skips what exists. ## 3. Send its mail Registration, password resets, email changes and invitations send mail from the kit's reactors. Until you set a mailer they print to the console, which is enough to try the kit. In production, pass an object with a `send(mail)` method that hands the mail to your provider: ```ts [TypeScript] import { useMailer } from "./kit/src/shared/mail.ts"; useMailer({ async send(mail) { await smtp.send({ to: mail.to, subject: mail.subject, text: mail.text }); } }); ``` ```python [Python] from kit.shared.mail import use_mailer class Smtp: def send(self, mail): smtp.send(to=mail.to, subject=mail.subject, text=mail.text) use_mailer(Smtp()) ``` ```rust [Rust] use kit::shared::mail::{use_mailer, Mail, Mailer}; struct Smtp; impl Mailer for Smtp { fn send(&self, mail: Mail) { /* hand mail.to, mail.subject, mail.text to your provider */ } } use_mailer(std::sync::Arc::new(Smtp)); ``` The texts are in the `reactors` files of `account` and `invitation`. When `send` throws, the reactor gets the event again, backing off; after 5 attempts it parks the event, and `kernable listeners retry` sends it once your provider is back (see [Failures and recovery](../concepts/failures)). A sent invitation is recorded as `InvitationMailSent`. ## 4. Grant the keys | Process | Grant | |---|---| | the kit's service | `run` on `tenancy.*`, `account.*`, `identity.*`, `group.*`, `invitation.*`; `listen`, `answer` and `check` on `*`; `vault_read` and `vault_erase`; `reserved_order=*` (its listeners run before yours); `privileged=bypass_authorization` (its own follow-up commands); `process=*` (it sets as whom each process acts) | | your API | `submit` and `query` on what it sends; `cache` on `account` (passkey challenges and OIDC state live there for a few minutes) | See [Keys per service](../how-to/keys) for issuing them. ## 5. Sign people in, in your API Your API is the one process that sees passwords, tokens and provider answers. `KitApi` holds what only it can do; wire it into your HTTP framework (`kit/example/server.ts` shows `node:http`, `kit/example/server.py` shows `http.server`, `kit/examples/server.rs` axum): ```ts [TypeScript] const api = new KitApi({ kernable, keys: signingKeysFrom("token-signing.pem"), service }); const signedIn = await api.signIn({ email, password, ip }); // access and refresh token const claims = api.check(accessToken); // per request, without the kernel const context = await api.contextFor(claims, tenantId); // senderAccount, senderSession, senderTenant, senderIdentity await kernable.submit(PlaceOrder.new({ … }), { aggregateId, tenant: tenantId, context }); ``` ```python [Python] api = KitApi(kernable, signing_keys_from("token-signing.pem"), service) signed_in = api.sign_in(email, password, ip=ip) # access and refresh token claims = api.check(access_token) # per request, without the kernel context = api.context_for(claims, tenant_id) # sender_account, sender_session, sender_tenant, sender_identity kernable.submit(PlaceOrder(...), aggregate_id=aggregate_id, tenant=tenant_id, context=context) ``` ```rust [Rust] let api = KitApi::new(kernable.clone(), signing_keys_from("token-signing.pem")?, &service_identity, &service_key_id); let signed_in = api.sign_in(&email, &password, ip, None).await?; // access and refresh token let claims = api.check(&access_token); // per request, without the kernel let context = api.context_for(&claims, Some(&tenant_id)).await?; // sender_account, sender_session, sender_tenant, sender_identity kernable.submit(Submit::command(PlaceOrder { .. }, &aggregate_id).tenant(&tenant_id).context(context)).await?; ``` In Python, `KitApi` blocks like the SDK's client; for FastAPI or another asyncio framework, `AsyncKitApi(KitApi(...))` has the same methods, awaited. In Rust, `KitApi` is async. Access tokens live 15 minutes and are checked by your API alone; the middleware still refuses an ended session, a suspended account or a revoked API key at once. Refresh tokens rotate, and using one twice ends its session. `KitApi` also verifies emails, resets passwords, sets up TOTP and passkeys, runs OAuth/OIDC (`oidcStart`, `oidcFinish`), checks API keys and accepts invitations. ## 6. Decide with groups and relations A tenant's owners may do everything in it. Everyone else gets permissions through groups, for the whole tenant or one aggregate: ```ts [TypeScript] CreateGroup.new({ name: "Customers", permissions: [ { pattern: "shop.PlaceOrder" }, { pattern: "shop.CancelOrder", when: "owner" }, // only orders they placed ] }); ``` ```python [Python] CreateGroup(name="Customers", permissions=[ {"pattern": "shop.PlaceOrder"}, {"pattern": "shop.CancelOrder", "when": "owner"}, # only orders they placed ]) ``` ```rust [Rust] CreateGroup { name: "Customers".into(), description: None, permissions: vec![ Permission { pattern: "shop.PlaceOrder".into(), ..Permission::default() }, Permission { pattern: "shop.CancelOrder".into(), when: Some("owner".into()), via_parent: None }, // only orders they placed ] }; ``` Nobody grants a group with permissions they do not hold. `Explain` answers why a request is allowed or refused, with the middleware's own steps. For lists, `visibility(readModel)` gives an app read model the same decision, so a list shows only what each asker may get. ## Next - [Your API: KitApi](./api) covers every sign-in method, tokens and API keys. - [Authorization](./authorization) explains each step of the middleware's decision. - [Recipes](./recipes) for common changes. ## Changing the kit Each domain has the same files: `domain`, `events`, `aggregates`, `commands`, `readmodels`, `queries`, `reactors` where it has any, and `index.ts`, `__init__.py` or `mod.rs`, which registers the parts. Events, commands and queries are classes, so a changed field shows wherever it is used. Settings you are likely to change (lockout, token lifetimes, reserved slugs, the passkey relying party) are in `shared/constants`. Fields are camelCase in TypeScript and snake_case in Python and Rust; the log holds camelCase either way. A newer kit is not installed over yours: copy it into another directory and take over what you want. --- # Tenancy A tenant is one customer organisation of your application: a company using your SaaS, a school, a shop. Everything that belongs to it (its identities, groups, invitations, and your own aggregates) lives under its tenant ID, so the kernel keeps tenants apart and can erase or move one as a whole. ## The tenant One aggregate, `tenancy.tenant`, whose ID *is* the tenant ID: a tenant `acme` lives under the tenant `acme`. It holds: | Field | What it is | |---|---| | `slug` | a short name for URLs and subdomains (`acme.example.com`): 3–63 lowercase letters, digits and single dashes, unique across tenants, not one of `RESERVED_SLUGS` | | `name` | the display name | | `status` | `pending`, `active`, `suspended` or `deleted` | | `owners` | the identities that own it; at least one once the first owner joined | | `suspension` | why it is suspended: `owner_requested`, `unpaid`, `abuse`, `by_platform` or `rejected` | | `mfaRequired` | whether everyone in it needs a second factor | | `signUp` | the platform tenant only: the sign-up policy | ### Lifecycle ```mermaid stateDiagram-v2 [*] --> pending: CreateTenant (self-service, waits for approval) [*] --> active: CreateTenant pending --> active: ActivateTenant pending --> suspended: SuspendTenant (rejected) active --> suspended: SuspendTenant suspended --> active: ReactivateTenant suspended --> deleted: DeleteTenant deleted --> [*] ``` - A **suspended** tenant keeps its data, but the middleware refuses its requests (`tenant_suspended`). The platform can still act on it; its owners can reactivate it only if they suspended it themselves (`owner_requested`). - **Deleting** is only possible from `suspended`, after `DELETE_AFTER_DAYS` (0 by default). It frees the slug, and the kit's reactors delete the tenant's identities and erase the emails of its invitations, both with their personal data; then every kit read model drops the tenant's rows. Your own read models should do the same on `tenancy.TenantDeleted`. - A rejected self-service tenant (`rejected`) cannot be reactivated. ## The platform tenant One tenant is special: the **platform tenant** (`PLATFORM_TENANT`, `platform` by default). It is where every account and every session lives, and where the people who run your application have their identities. Its groups hold their permissions **in every tenant**, and its owners hold every permission everywhere. It cannot be suspended or deleted. `bootstrap()` creates it. Its tenant also holds the **sign-up policy**, changed with `ChangeSignUpPolicy`: | Setting | Values | Effect | |---|---|---| | `accounts` | `off`, `open` | `open`: anyone may register. `off`: accounts come only from invitations | | `tenants` | `off`, `pending`, `active` | whether a signed-in person may create a tenant themselves, and whether it starts `active` or waits as `pending` for the platform's `ActivateTenant` | ## Owners Owners are kept on the tenant itself, nowhere else. They hold every permission in their tenant and appear in group lists as the virtual group `owners`, which cannot be deleted. The last owner cannot be removed (`last_owner`). How the first owner arrives depends on who creates the tenant: - **Self-service**: a signed-in person sends `CreateTenant` with `owner: { accountId }` (their own account). The kit creates their identity in the new tenant and adds it as owner. - **By the platform**: `CreateTenant` with `owner: { email }`. The kit sends an owner invitation to that address; accepting it adds the person as owner. An identity that owns its tenant cannot be deleted; remove it as owner first. The kernel checks this atomically through the tenant's `owner` key. ## Requiring a second factor `RequireMfa` with `required: true` makes the middleware refuse every request in the tenant from a session without a second factor (`mfa_required`): TOTP or a recovery code proved in this session, or a passkey, which counts as two factors. API keys of service identities are not affected. ## Settings In `kit/src/shared/constants.ts` (Python: `kit/shared/constants.py`, Rust: `src/shared/constants.rs`): | Constant | Default | What it does | |---|---|---| | `PLATFORM_TENANT` | `platform` | the platform tenant's ID | | `RESERVED_SLUGS` | `platform`, `www`, `api`, `admin`, `app`, `mail`, `auth`, `login`, `static` | slugs nobody may take | | `DELETE_AFTER_DAYS` | `0` | how long a suspended tenant waits before it may be deleted | ## Reference ### Types The types the tables below name, from `tenancy/domain.ts`: ```ts type TenantStatus = "none" | "pending" | "active" | "suspended" | "deleted"; type SuspensionReason = "owner_requested" | "unpaid" | "abuse" | "by_platform" | "rejected"; interface SignUpPolicy { /** The status a self-service tenant starts with, or `off`. */ tenants: "off" | "pending" | "active"; accounts: "off" | "open"; } type Owner = { email: string } | { accountId: string }; ``` ### Aggregates | Aggregate | Unique keys | Facts | |---|---|---| | `tenancy.tenant` | `slug` (global), `owner` | `active`, `exists` | ### Commands Secret fields leave the vault once the command completes; personal fields stay in the vault until their subject is erased. | Command | Fields | Writes | Who may send it | Refused with | |---|---|---|---|---| | `ActivateTenant`
on `tenant` | — | `TenantActivated` | the platform | `invalid_state` | | `AddOwner`
on `tenant` | `identityId: string` | `OwnerAdded` | owners | `exists`, `invalid_identity`, `not_found`; requires: identity.identity is active | | `ChangeSignUpPolicy`
on `tenant` | `tenants: "off" \| "pending" \| "active"`
`accounts: "off" \| "open"` | `SignUpPolicyChanged` | the platform | `invalid_policy`, `not_found`, `platform` | | `ChangeTenantSlug`
on `tenant` | `slug: string` | `TenantSlugChanged` | a group permission | `invalid_slug`, `invalid_state` | | `CreateTenant`
on `tenant` | `slug: string`
`name: string`
`owner: Owner`
`status?: "active" \| "pending"` | `TenantCreated` | any signed-in account (self-service; the platform's groups otherwise) | `exists`, `invalid_name`, `invalid_owner`, `invalid_slug`, `invalid_tenant` | | `DeleteTenant`
on `tenant` | — | `TenantDeleted` | the platform | `invalid_state`, `platform`, `too_early` | | `ReactivateTenant`
on `tenant` | — | `TenantReactivated` | the platform (owners too, after owner_requested) | `invalid_state` | | `RemoveOwner`
on `tenant` | `identityId: string` | `OwnerRemoved` | owners | `last_owner`, `not_found` | | `RenameTenant`
on `tenant` | `name: string` | `TenantRenamed` | a group permission | `invalid_name`, `invalid_state` | | `RequireMfa`
on `tenant` | `required: boolean` | `MfaRequirementChanged` | owners | `invalid_state` | | `SuspendTenant`
on `tenant` | `reason: SuspensionReason` | `TenantSuspended` | the platform (owners too, with owner_requested) | `invalid_reason`, `invalid_state`, `platform` | ### Events | Event | Fields | Vault | Note | |---|---|---|---| | `TenantCreated` | `slug: string`
`name: string`
`owner: Owner`
`status: "active" \| "pending"` | — | — | | `TenantActivated` | — | — | — | | `TenantRenamed` | `name: string` | — | — | | `TenantSlugChanged` | `slug: string` | — | — | | `TenantSuspended` | `reason: SuspensionReason`
`at: number` | — | — | | `TenantReactivated` | — | — | — | | `TenantDeleted` | — | — | — | | `OwnerAdded` | `identityId: string` | — | — | | `OwnerRemoved` | `identityId: string` | — | — | | `MfaRequirementChanged` | `required: boolean` | — | — | | `SignUpPolicyChanged` | `tenants: "off" \| "pending" \| "active"`
`accounts: "off" \| "open"` | — | — | ### Queries A query about one aggregate names it (`static target`): the middleware decides on that aggregate as on a command's. | Query | Fields | About | Answer | Who may ask | |---|---|---|---|---| | `GetTenantBySlug`
by `tenancy.tenants` | `slug: string` | — | `{ id: string; name: string; status: TenantStatus }` | the API (its `api` group) | | `GetSignUpPolicy`
by `tenancy.tenants` | — | — | `SignUpPolicy { tenants, accounts }` | the API (its `api` group) | | `GetTenant`
by `tenancy.tenants` | `tenantId: string` | `tenancy.tenant` in `tenantId` | `TenantRow { id, slug, name, status, suspension, mfaRequired, owners }` | a group permission | | `ListTenants`
by `tenancy.tenants` | `status?: TenantStatus`
`after?: string`
`limit?: number` | — | `{ tenants: TenantRow[]; more: boolean }` | the platform | ### Listeners | Listener | On | Submits | |---|---|---| | `tenancy.tenants` (read model, order -20) | `TenantCreated`, `TenantActivated`, `TenantRenamed`, `TenantSlugChanged`, `TenantSuspended`, `TenantReactivated`, `TenantDeleted`, `OwnerAdded`, `OwnerRemoved`, `MfaRequirementChanged`, `SignUpPolicyChanged` | — | --- # Accounts and sessions An account is a person who can sign in: one email address, one password (or none), their second factors and linked logins. It is the same account in every tenant; what the person may do in a tenant comes from their [identity](./identity) there. Every account lives in the [platform tenant](./tenancy#the-platform-tenant). The `account` domain has three aggregates: | Aggregate | One per | Lives under | |---|---|---| | `account.account` | person | the platform tenant | | `account.session` | sign-in; ends on sign-out, expiry or revocation | the platform tenant | | `account.sso` | single sign-on configuration of a tenant | that tenant | Most of the commands here are sent by your API through [`KitApi`](./api), which picks session IDs, generates tokens and checks provider answers. The page explains what happens underneath. ## Registration and email verification ```mermaid sequenceDiagram participant P as person participant A as your API (KitApi) participant K as kernel participant M as account.mail P->>A: register(email, name, password) A->>K: RegisterAccount K-->>M: AccountRegistered, VerificationRequested M->>P: mail with a token P->>A: verifyEmail(token) A->>K: VerifyEmail K-->>A: EmailVerified (account active) ``` - An account starts `pending` and becomes `active` with `VerifyEmail`. The token is valid for `VERIFICATION_HOURS` (48) and is erased once used. - **Registration does not reveal who has an account.** If the address is taken, `register` answers exactly as on success. The owner of the address gets a "you already have an account" mail instead (`RegistrationAttempted`). - A registration that expired unverified does not block the address: registering again takes it over with a new token. - Whether anyone may register is the platform's [sign-up policy](./tenancy#the-platform-tenant). With `accounts: off`, accounts come only from [invitations](./invitation); an invitation proves the email, so such an account is active at once. ## Signing in `SignIn` checks the password against its argon2id hash and writes either `SignedIn` or `SignInFailed` with a reason: `wrong_password`, `locked`, `not_verified` or `not_active`. A missing account answers `wrong_password` too. **Lockout:** after `LOCKOUT_AFTER` (5) failures in a row, the account is locked for 1 minute, then 2, 4 and so on, at most `LOCKOUT_MAX_MINUTES` (15). The lock is computed from the failures, not stored, and a successful sign-in resets it. Failed second factors count as well. After `SignedIn`, a reactor opens the **session** (`OpenSession`). Your API then holds two tokens: | Token | Lifetime | Checked by | |---|---|---| | access token (EdDSA, claims: account, session, factors) | `ACCESS_TOKEN_MINUTES` (15) | your API alone, per request, without the kernel | | refresh token `..` | as long as the session | the kernel (`RefreshSession`) | **Refresh tokens rotate.** Each `RefreshSession` issues the next generation. A token from an older generation is a reuse, a sign that it was stolen, so the session ends at once (`reuse`). Sessions also end after `SESSION_IDLE_DAYS` (14) without a refresh, and after `SESSION_MAX_DAYS` (90) in any case. A short-lived access token does not leave a gap: the middleware learns from the events when a session ends, an account is suspended or a password changes, and refuses that session's requests at once. ## Second factors | Factor | How it is set up | Counts as | |---|---|---| | TOTP (an authenticator app) | `StartTotp` with a secret your API generated, then `ConfirmTotp` with the first code; it also creates 10 recovery codes, shown once | the second factor | | recovery code | from `ConfirmTotp` or `RegenerateRecoveryCodes`; each works once | the second factor | | passkey (WebAuthn) | `AddPasskey` with the browser's attestation | two factors: it proves possession and the user | An account with TOTP on needs it in **every** session: until `ProveSecondFactor` (or `ProvePasskey`) succeeds, the middleware allows nothing but signing out. A [tenant may require](./tenancy#requiring-a-second-factor) a second factor from everyone in it. Passkeys can also be the only sign-in method: `SignInWithPasskey` needs no password. Passkeys are bound to your site: set the relying party with `usePasskeys({ rpId, rpName, origins })`. ## Other ways to sign in - **Social logins** (Google, GitHub, any OpenID Connect provider): configured in `KitApi`. A login that is linked to an account signs in to it. A login whose email nobody has registers a new person. A login whose email belongs to an account that has not linked it is refused (`not_linked`): the person signs in another way and links it (`LinkProvider`), so nobody takes over an account through a provider. - **Single sign-on per tenant** (`account.sso`): a tenant's owners configure their identity provider (`CreateSso`) and add their email domains. A domain counts once its DNS carries the TXT record `_kernable-sso.` (`VerifySsoDomain`, run by your API); each domain belongs to one tenant only. People with an address in a verified domain then sign in through that provider, and their first SSO login is linked to the account with the same email. - **Enforced SSO** (`EnforceSso`): in that tenant, only sessions signed in through its SSO count. Owners are exempt, so a broken SSO configuration can be mended. The same person may still sign in with a password for *other* tenants. ## Changing an account | Change | Command | What else happens | |---|---|---| | password | `ChangePassword` (with the current one, if there is one) | every other session of the account ends; this one stays | | forgotten password | `RequestPasswordReset`, then `ResetPassword` with the mailed token (valid `RESET_MINUTES`, 60) | every session ends | | email | `RequestEmailChange`, then `ConfirmEmailChange` with the token mailed to the new address | the old address stays until it is confirmed | Who may do these: the account itself, or the platform. A person may suspend their own account (`owner_requested`); suspending for other reasons, reactivating and deleting are the platform's. **Deleting an account** (`DeleteAccount`, from `suspended`) erases its personal data from every event of the account and its sessions, and deletes its identities in every tenant. An identity that still owns a tenant stops this: its deletion parks for an operator until another owner is found. ## Settings | Constant | Default | What it does | |---|---|---| | `LOCKOUT_AFTER`, `LOCKOUT_MAX_MINUTES` | 5, 15 | failures before the lock, and its longest duration | | `VERIFICATION_HOURS` | 48 | how long an email verification token is valid | | `RESET_MINUTES` | 60 | how long a password reset token is valid | | `ACCESS_TOKEN_MINUTES` | 15 | the access token's lifetime | | `SESSION_IDLE_DAYS`, `SESSION_MAX_DAYS` | 14, 90 | when a session ends without a refresh, and at the latest | | `PASSKEYS` | `localhost` | the relying party; set it with `usePasskeys` | | `CHALLENGE_SECONDS` | 300 | how long a passkey challenge is valid | The password hash parameters (argon2id, 19 MiB, 2 iterations, OWASP's recommendation) are in `shared/secrets`. ## Reference ### Types The types the tables below name, from `account/domain.ts`: ```ts type AccountStatus = "none" | "pending" | "active" | "suspended" | "deleted"; type AccountSuspensionReason = "owner_requested" | "abuse" | "by_platform"; type TokenPurpose = "verify" | "reset" | "email_change"; type SessionEnd = "signed_out" | "revoked" | "account_suspended" | "password_changed" | "account_deleted" | "reuse" | "idle" | "expired"; ``` ### Aggregates | Aggregate | Unique keys | Facts | |---|---|---| | `account.account` | `link`, `email` | `active`, `mfa` | | `account.session` | — | — | | `account.sso` | `domain` (global) | — | ### Commands Secret fields leave the vault once the command completes; personal fields stay in the vault until their subject is erased. | Command | Fields | Writes | Who may send it | Refused with | |---|---|---|---|---| | `AddSessionFactor`
on `session` | `factor: string`
`accountId: string` | `SessionFactorAdded` | only the kit's reactors | `invalid_session`, `invalid_state` | | `EndSession`
on `session` | `reason?: SessionEnd` | `SessionEnded` | a group permission, or the sender itself | `invalid_state` | | `OpenSession`
on `session` | `accountId: string`
`method: string`
`factors: string[]`
`refreshTokenHash: string`
`ip: string \| null` (secret)
`userAgent: string \| null` (secret)
`at: number` | `SessionOpened` | only the kit's reactors | `exists` | | `RefreshSession`
on `session` | `refreshToken: string` (secret)
`newRefreshToken: string` (secret)
`ip?: string` (secret) | `SessionEnded`, `SessionRefreshed` | anyone (the request carries its proof) | `invalid_token` | | `RegisterAccount`
on `account` | `email: string` (personal)
`name: string` (personal)
`password?: string` (secret)
`verified?: boolean` | `AccountRegistered`, `VerificationRequested` | anyone (the request carries its proof) (`verified` only from the API) | `exists`, `invalid_email`, `invalid_name`, `weak_password` | | `RequestVerification`
on `account` | — | `VerificationRequested` | the API (its `api` group) | `invalid_state` | | `SignIn`
on `account` | `sessionId: string`
`password: string` (secret)
`refreshToken: string` (secret)
`ip?: string` (secret)
`userAgent?: string` (secret) | `SignInFailed`, `SignedIn` | anyone (the request carries its proof) | `invalid_token`, `not_found` | | `VerifyEmail`
on `account` | `token: string` (secret) | `EmailVerified` | anyone (the request carries its proof) | `expired_token`, `invalid_token` | | `SuspendAccount`
on `account` | `reason: AccountSuspensionReason` | `AccountSuspended` | the platform (the person too, with owner_requested) | `invalid_reason`, `invalid_state` | | `ReactivateAccount`
on `account` | — | `AccountReactivated` | the platform | `invalid_state` | | `DeleteAccount`
on `account` | — | `AccountDeleted` | the platform | `invalid_state` | | `ChangePassword`
on `account` | `current?: string` (secret)
`new: string` (secret)
`keepSessionId?: string` | `PasswordChanged` | a group permission, or the sender itself | `invalid_state`, `weak_password`, `wrong_password` | | `RequestPasswordReset`
on `account` | — | `PasswordResetRequested` | the API (its `api` group) | `invalid_state` | | `ResetPassword`
on `account` | `token: string` (secret)
`new: string` (secret) | `PasswordReset` | anyone (the request carries its proof) | `expired_token`, `invalid_token`, `weak_password` | | `RequestEmailChange`
on `account` | `email: string` (personal) | `EmailChangeRequested` | a group permission, or the sender itself | `invalid_email`, `invalid_state`, `unchanged` | | `ConfirmEmailChange`
on `account` | `token: string` (secret) | `EmailChanged` | anyone (the request carries its proof) | `expired_token`, `invalid_token` | | `NoteRegistrationAttempt`
on `account` | — | `RegistrationAttempted` | the API (its `api` group) | `invalid_state` | | `StartTotp`
on `account` | `secret: string` (personal) | `TotpStarted` | a group permission, or the sender itself | `exists`, `invalid_secret`, `invalid_state` | | `ConfirmTotp`
on `account` | `code: string` (secret)
`recoveryCodes: string[]` (secret) | `TotpConfirmed`, `RecoveryCodesGenerated` | a group permission, or the sender itself | `invalid_recovery_codes`, `invalid_state`, `wrong_code` | | `DisableTotp`
on `account` | `code: string` (secret) | `TotpDisabled` | a group permission, or the sender itself | `wrong_code` | | `RegenerateRecoveryCodes`
on `account` | `code: string` (secret)
`recoveryCodes: string[]` (secret) | `RecoveryCodesGenerated` | a group permission, or the sender itself | `invalid_recovery_codes`, `wrong_code` | | `ProveSecondFactor`
on `account` | `sessionId: string`
`code: string` (secret) | `SecondFactorFailed`, `SecondFactorProved` | anyone (the request carries its proof) | `invalid_state` | | `AddPasskey`
on `account` | `challenge: string`
`response: RegistrationResponseJSON`
`label?: string` (personal) | `PasskeyAdded` | a group permission, or the sender itself | `exists`, `invalid_passkey`, `invalid_state` | | `RemovePasskey`
on `account` | `credentialId: string` | `PasskeyRemoved` | a group permission, or the sender itself | `last_method`, `not_found` | | `SignInWithPasskey`
on `account` | `sessionId: string`
`challenge: string`
`response: AuthenticationResponseJSON`
`refreshToken: string` (secret)
`ip?: string` (secret)
`userAgent?: string` (secret) | `SignInFailed`, `PasskeyUsed`, `SignedIn` | the API (its `api` group) | `invalid_token`, `not_found` | | `ProvePasskey`
on `account` | `sessionId: string`
`challenge: string`
`response: AuthenticationResponseJSON` | `SecondFactorFailed`, `PasskeyUsed`, `SecondFactorProved` | the API (its `api` group) | `invalid_state` | | `SignInWithProvider`
on `account` | `sessionId: string`
`provider: string`
`subject: string` (secret)
`email?: string` (secret)
`emailVerified?: boolean`
`autoLink?: boolean`
`refreshToken: string` (secret)
`ip?: string` (secret)
`userAgent?: string` (secret) | `SignInFailed`, `SignedIn`, `ProviderLinked` | the API (its `api` group) | `invalid_provider`, `invalid_subject`, `invalid_token`, `not_found` | | `LinkProvider`
on `account` | `provider: string`
`subject: string` (secret)
`email?: string` (secret) | `ProviderLinked` | the API (its `api` group) | `exists`, `invalid_provider`, `invalid_state`, `invalid_subject` | | `UnlinkProvider`
on `account` | `provider: string`
`subject: string` (secret) | `ProviderUnlinked` | a group permission, or the sender itself | `last_method`, `not_found` | | `CreateSso`
on `sso` | `issuer: string`
`clientId: string`
`clientSecret: string` (secret) | `SsoCreated` | owners | `exists`, `invalid_client`, `invalid_issuer` | | `UpdateSso`
on `sso` | `issuer: string`
`clientId: string`
`clientSecret: string` (secret) | `SsoUpdated` | owners | `invalid_issuer`, `not_found` | | `AddSsoDomain`
on `sso` | `domain: string` | `SsoDomainAdded` | owners | `exists`, `invalid_domain`, `not_found` | | `VerifySsoDomain`
on `sso` | `domain: string`
`found: string[]` | `SsoDomainVerified` | the API (its `api` group) | `not_found`, `not_verified` | | `RemoveSsoDomain`
on `sso` | `domain: string` | `SsoDomainRemoved` | owners | `not_found` | | `EnforceSso`
on `sso` | `enforced: boolean` | `SsoEnforcementChanged` | owners | `invalid_state` | | `DisableSso`
on `sso` | — | `SsoEnforcementChanged`, `SsoDisabled` | owners | `invalid_state` | | `EnableSso`
on `sso` | — | `SsoEnabled` | owners | `invalid_state` | | `DeleteSso`
on `sso` | — | `SsoDeleted` | owners | `not_found` | ### Events | Event | Fields | Vault | Note | |---|---|---|---| | `AccountRegistered` | `email: string`
`name: string`
`passwordHash: string \| null`
`verified: boolean`
`at: number` | `email`, `name`, `passwordHash` under `the aggregate` | — | | `VerificationRequested` | `tokenId: string`
`token: string \| null`
`email: string \| null`
`expiresAt: number` | `email` under `the aggregate`; `token` under `token:` | The token stays in the vault under its own subject until it is used. | | `EmailVerified` | `tokenId: string` | erases `token:` | Erases the used token. | | `SignedIn` | `sessionId: string`
`method: string`
`factors: string[]`
`refreshTokenHash: string`
`ip: string \| null`
`userAgent: string \| null`
`mfa: boolean`
`at: number` | `ip`, `userAgent` under `the aggregate` | — | | `SignInFailed` | `method: string`
`reason: "wrong_password" \| "locked" \| "not_verified" \| "not_active" \| "not_linked"`
`ip: string \| null`
`at: number` | `ip` under `the aggregate` | An event, not a rejection: audited, and it counts toward the lockout. | | `SessionOpened` | `accountId: string`
`method: string`
`factors: string[]`
`refreshTokenHash: string`
`ip: string \| null`
`userAgent: string \| null`
`at: number` | `ip`, `userAgent` under `the accountId` | — | | `SessionRefreshed` | `generation: number`
`refreshTokenHash: string`
`ip: string \| null`
`accountId: string`
`factors: string[]`
`at: number` | `ip` under `the accountId` | — | | `SessionFactorAdded` | `factor: string` | — | — | | `SessionEnded` | `reason: SessionEnd` | — | — | | `AccountSuspended` | `reason: AccountSuspensionReason`
`at: number` | — | Ends every session (a reactor); sign-in is refused. | | `AccountReactivated` | — | — | — | | `AccountDeleted` | — | erases `the aggregate` | Erases the account's personal data, in its events and its sessions'. | | `PasswordChanged` | `hash: string`
`keepSessionId: string \| null`
`at: number` | `hash` under `the aggregate` | Ends the account's other sessions (a reactor); `keepSessionId` is the one it was changed from. | | `PasswordResetRequested` | `tokenId: string`
`token: string \| null`
`email: string \| null`
`expiresAt: number` | `email` under `the aggregate`; `token` under `token:` | The reset token sits in the vault under its own subject until used. | | `PasswordReset` | `tokenId: string`
`hash: string`
`at: number` | `hash` under `the aggregate`; erases `token:` | Erases the used token; ends every session (a reactor). | | `EmailChangeRequested` | `tokenId: string`
`token: string \| null`
`email: string \| null`
`expiresAt: number` | `email` under `the aggregate`; `token` under `token:` | The new address gets a token; it holds nothing until confirmed. | | `EmailChanged` | `tokenId: string`
`email: string` | `email` under `the aggregate`; erases `token:` | Erases the used token; the account now holds the new address. | | `RegistrationAttempted` | `email: string \| null`
`at: number` | `email` under `the aggregate` | Someone registered with this account's address: it gets a mail saying it exists. | | `TotpStarted` | `secret: string \| null` | `secret` under `the aggregate` | The TOTP secret the API generated and showed: personal, in the vault. Not in use until confirmed. | | `TotpConfirmed` | `step: number`
`at: number` | — | TOTP is the account's second factor from now on (the first code's step: no code twice). | | `TotpDisabled` | — | — | — | | `RecoveryCodesGenerated` | `hashes: string[]` | — | The recovery codes' hashes; the codes were shown once. | | `SecondFactorProved` | `sessionId: string`
`method: "totp" \| "recovery" \| "passkey"`
`step: number \| null`
`recoveryCodeHash: string \| null` | — | A session proved a second factor; a reactor adds it to the session. | | `SecondFactorFailed` | `sessionId: string`
`at: number` | — | Counts toward the lockout, like a wrong password. | | `PasskeyAdded` | `credentialId: string \| null`
`publicKey: string`
`counter: number`
`label: string \| null`
`at: number` | `credentialId`, `label` under `the aggregate` | A passkey: its credential ID and label are personal; the public key (COSE, base64url) is not. | | `PasskeyRemoved` | `credentialId: string \| null` | `credentialId` under `the aggregate` | — | | `PasskeyUsed` | `credentialId: string \| null`
`counter: number` | `credentialId` under `the aggregate` | A passkey signed: its counter moves on (a cloned authenticator would fall behind). | | `ProviderLinked` | `provider: string`
`subject: string \| null`
`email: string \| null`
`at: number` | `subject`, `email` under `the aggregate` | An external login (OAuth/OIDC, or a tenant's SSO as `sso:`) linked to the account. | | `ProviderUnlinked` | `provider: string`
`subject: string \| null` | `subject` under `the aggregate` | — | | `SsoCreated` | `issuer: string`
`clientId: string`
`clientSecret: string \| null` | `clientSecret` under `sso:` | The client secret is personal data of the configuration (`sso:`), erased with it. | | `SsoUpdated` | `issuer: string`
`clientId: string`
`clientSecret: string \| null` | `clientSecret` under `sso:` | — | | `SsoDomainAdded` | `domain: string`
`txt: string` | — | `txt`: the value the domain's DNS must carry at `_kernable-sso.`. | | `SsoDomainVerified` | `domain: string` | — | — | | `SsoDomainRemoved` | `domain: string` | — | — | | `SsoEnforcementChanged` | `enforced: boolean` | — | Enforced: in this tenant, only sessions signed in through this SSO count (owners excepted). | | `SsoDisabled` | — | — | — | | `SsoEnabled` | — | — | — | | `SsoDeleted` | — | erases `sso:` | — | ### Queries A query about one aggregate names it (`static target`): the middleware decides on that aggregate as on a command's. | Query | Fields | About | Answer | Who may ask | |---|---|---|---|---| | `FindAccountByEmail`
by `account.accounts` | `email: string` | — | `{ id: string; status: AccountStatus }` | the API (its `api` group) | | `GetAccount`
by `account.accounts` | `accountId: string` | `account.account` in `accountId` | `AccountRow { id, email, name, status }` | the API (its `api` group), or the sender itself | | `ListSessions`
by `account.accounts` | `accountId: string` | `account.account` in `accountId` | `SessionRow[]` | a group permission, or the sender itself | | `FindAccountByLink`
by `account.accounts` | `provider: string`
`subject: string` | — | `{ id: string; status: AccountStatus }` | the API (its `api` group) | | `FindSsoByDomain`
by `account.ssos` | `domain: string` | — | `{ tenant: string; ssoId: string; issuer: string; clientId: string; enforced: boolean }` | the API (its `api` group) | | `GetSsoSecret`
by `account.ssos` | `ssoId: string` | — | `{ issuer: string; clientId: string; clientSecret: string \| null }` | the API (its `api` group) | | `GetSso`
by `account.ssos` | `ssoId: string` | — | `Omit` | owners | ### Listeners | Listener | On | Submits | |---|---|---| | `account.accounts` (read model, order -20) | `AccountRegistered`, `EmailVerified`, `EmailChanged`, `ProviderLinked`, `ProviderUnlinked`, `AccountSuspended`, `AccountReactivated`, `AccountDeleted`, `SessionOpened`, `SessionRefreshed`, `SessionEnded` | — | | `account.ssos` (read model, order -20) | `SsoCreated`, `SsoUpdated`, `SsoDomainAdded`, `SsoDomainVerified`, `SsoDomainRemoved`, `SsoEnforcementChanged`, `SsoDisabled`, `SsoEnabled`, `SsoDeleted`, `TenantDeleted` | — | | `account.mail` (reactor, order -10) | `VerificationRequested` | side effects only | | `account.mail` (reactor, order -10) | `PasswordResetRequested` | side effects only | | `account.mail` (reactor, order -10) | `EmailChangeRequested` | side effects only | | `account.mail` (reactor, order -10) | `RegistrationAttempted` | side effects only | | `account.sessions` (reactor, order -10) | `SecondFactorProved` | `AddSessionFactor` | | `account.sessions` (reactor, order -10) | `SignedIn` | `OpenSession` | | `account.ending` (reactor, order -10) | `AccountSuspended` | side effects only | | `account.ending` (reactor, order -10) | `AccountDeleted` | side effects only | | `account.ending` (reactor, order -10) | `PasswordChanged` | side effects only | | `account.ending` (reactor, order -10) | `PasswordReset` | side effects only | --- # Identities An identity is someone *inside one tenant*: a person (through their [account](./account)) or a service, such as another program that calls your API. Permissions, profiles and group memberships belong to identities, so the same person can own one tenant and be a guest in another, with a different display name in each. One aggregate, `identity.identity`, with a random ID, under its tenant. | Field | What it is | |---|---| | `kind` | `person`, `service`, or `process` (a business process run by the kernel) | | `accountId` | the account, for persons; at most one identity per account and tenant | | `profile` | `displayName`, `title`, `email`: personal data, in the vault under `identity:` | | `memberships` | the groups it belongs to: each with an ID, the group, a scope and an optional expiry, `active` or `suspended` | | `apiKeys` | services and processes only: the hashes of their keys, with label, expiry and whether revoked | | `status` | `active`, `suspended` or `deleted` | ## How identities come to be You rarely create a person's identity yourself; the kit's reactors do it: - **Invitation accepted**: the person gets an identity in the tenant (or keeps theirs) with the invitation's groups, and becomes an owner for an owner invitation. - **Self-service tenant**: its creator gets an identity in it and becomes its owner. Services are created with `CreateServiceIdentity` by anyone whose group allows it. ## Memberships `GrantMembership` puts an identity in a group: | Scope | Means | |---|---| | none | the group's permissions apply in the whole tenant | | `{ type: "shop.order", id: "o-17" }` | they apply only to that one aggregate, like a relation set by hand | A membership can **expire** (`expiresAt`): it stops counting at that moment, with no command needed. It can be suspended and resumed without losing it, or revoked. When a group is deleted its memberships stop counting at once, and a reactor revokes them afterwards. Nobody grants a group whose permissions they do not hold themselves (see [no escalation](./authorization#no-escalation)). ## API keys A service identity signs its requests with an API key instead of a session: 1. `KitApi.createApiKey(...)` generates the key, sends `CreateApiKey` with it, and returns it **once**: `key__`. Only its hash is stored. 2. A request carries the key; your API calls `KitApi.checkApiKey(key)`, which compares the hash and returns the request context to submit with (the key's identity and tenant). The answer is cached for a minute. 3. `RevokeApiKey` ends it. The middleware learns of it from the event and refuses the key at once, cache or not. Every command sent with a key carries the key's ID in `senderSession`, so the log shows each use; there is no separate event per use. ## Processes A [business process](../concepts/processes) the kernel runs sends commands like a service does, so it needs an identity too. The kit creates one for each process on its first registration: in the platform tenant, kind `process`, named after the process (`shop.fulfilment`), with an API key, and tells the kernel to act as that identity. The new identity holds **no permissions**. A platform administrator grants it a group of the platform tenant with exactly what the process sends (`shop.Confirm`, `shop.Ship`, `shop.Cancel`); platform groups hold in every tenant. Until then its commands are refused and its cases stop as incidents, which `kernable processes retry` resumes once the group is granted. Revoking the process's API key stops the process at once, without a deploy. A later deploy does not give it a new key: resuming it is a deliberate step (a new key, then `processAct` with it). ## Leaving and deleting - An identity may **leave** its tenant itself (`DeleteIdentity` on its own identity). - Others with the permission suspend, reactivate and delete identities. - An identity that **owns** its tenant cannot be deleted until it is removed as owner; the kernel checks this atomically. - Deleting erases the profile from the vault. When a tenant or an account is deleted, the kit's reactors delete the identities concerned. ## Reference ### Types The types the tables below name, from `identity/domain.ts`: ```ts type IdentityKind = "person" | "service" | "process"; type IdentityStatus = "none" | "active" | "suspended" | "deleted"; interface Scope { type: string; id: string } interface Membership { membershipId: string; group: string; scope: Scope | null; expiresAt: number | null; status: "active" | "suspended"; } interface ApiKey { keyId: string; hash: string; label: string; expiresAt: number | null; revoked: boolean } ``` ### Aggregates | Aggregate | Unique keys | Facts | |---|---|---| | `identity.identity` | `account`, `process` | `active` | ### Commands Secret fields leave the vault once the command completes; personal fields stay in the vault until their subject is erased. | Command | Fields | Writes | Who may send it | Refused with | |---|---|---|---|---| | `CreateApiKey`
on `identity` | `keyId: string`
`secret: string` (secret)
`label?: string`
`expiresAt?: number \| null` | `ApiKeyCreated` | a group permission | `exists`, `invalid_key`, `invalid_state`, `not_a_service` | | `CreateServiceIdentity`
on `identity` | `displayName: string` (personal) | `IdentityCreated` | a group permission | `exists`, `invalid_name` | | `CreateProcessIdentity`
on `identity` | `process: string` | `IdentityCreated` | only the kit's reactors | `exists`, `invalid_process` | | `CreateIdentity`
on `identity` | `accountId: string`
`displayName?: string` (personal)
`title?: string` (personal)
`email?: string` (personal) | `IdentityCreated` | only the kit's reactors | `exists`, `invalid_account` | | `DeleteIdentity`
on `identity` | — | `IdentityDeleted` | a group permission, or the sender itself | `not_found`; requires: no tenancy.tenant holds it as owner | | `GrantMembership`
on `identity` | `membershipId: string`
`group: string`
`scope?: Scope \| null`
`expiresAt?: number \| null` | `MembershipGranted` | a group permission; only with permissions the sender holds | `exists`, `invalid_membership`, `invalid_scope`, `invalid_state`; requires: group.group is active | | `ReactivateIdentity`
on `identity` | — | `IdentityReactivated` | a group permission | `invalid_state` | | `ResumeMembership`
on `identity` | `membershipId: string` | `MembershipResumed` | a group permission | `not_found` | | `RevokeApiKey`
on `identity` | `keyId: string` | `ApiKeyRevoked` | a group permission | `not_found` | | `RevokeMembership`
on `identity` | `membershipId: string` | `MembershipRevoked` | a group permission | `not_found` | | `SuspendMembership`
on `identity` | `membershipId: string` | `MembershipSuspended` | a group permission | `not_found` | | `SuspendIdentity`
on `identity` | `reason?: string` | `IdentitySuspended` | a group permission | `invalid_state` | | `UpdateProfile`
on `identity` | `displayName?: string` (personal)
`title?: string` (personal)
`email?: string` (personal) | `ProfileUpdated` | a group permission, or the sender itself | `invalid_state` | ### Events | Event | Fields | Vault | Note | |---|---|---|---| | `IdentityCreated` | `kind: IdentityKind`
`accountId: string \| null`
`displayName: string \| null`
`title: string \| null`
`email: string \| null` | `displayName`, `title`, `email` under `identity:` | — | | `ProfileUpdated` | `displayName: string \| null`
`title: string \| null`
`email: string \| null` | `displayName`, `title`, `email` under `identity:` | — | | `MembershipGranted` | `membershipId: string`
`group: string`
`scope: Scope \| null`
`expiresAt: number \| null` | — | — | | `MembershipSuspended` | `membershipId: string` | — | — | | `MembershipResumed` | `membershipId: string` | — | — | | `MembershipRevoked` | `membershipId: string` | — | — | | `ApiKeyCreated` | `keyId: string`
`hash: string`
`label: string`
`expiresAt: number \| null` | — | — | | `ApiKeyRevoked` | `keyId: string` | — | — | | `IdentitySuspended` | `reason: string` | — | — | | `IdentityReactivated` | — | — | — | | `IdentityDeleted` | — | erases `identity:` | Erases the identity's personal data. | ### Queries A query about one aggregate names it (`static target`): the middleware decides on that aggregate as on a command's. | Query | Fields | About | Answer | Who may ask | |---|---|---|---|---| | `FindIdentityByAccount`
by `identity.identities` | `accountId: string` | — | `{ identityId: string; status: IdentityStatus }` | the API (its `api` group) | | `GetIdentity`
by `identity.identities` | `identityId: string` | `identity.identity` in `identityId` | `IdentityRow { id, tenant, kind, accountId, displayName, status, memberships, apiKeys }` | a group permission, or the sender itself | | `ListMyIdentities`
by `identity.identities` | `accountId: string` | `account.account` in `accountId` | `{ tenant: string; identityId: string; status: IdentityStatus }[]` | any signed-in account, or the sender itself | | `ListIdentities`
by `identity.identities` | — | — | `IdentityRow[]` | a group permission | | `FindApiKey`
by `identity.identities` | `keyId: string` | — | `{ tenant: string; identityId: string; hash: string; expiresAt: number \| null; revoked: boolean }` | the API (its `api` group) | ### Listeners | Listener | On | Submits | |---|---|---| | `identity.identities` (read model, order -20) | `IdentityCreated`, `ProfileUpdated`, `MembershipGranted`, `MembershipSuspended`, `MembershipResumed`, `MembershipRevoked`, `ApiKeyCreated`, `ApiKeyRevoked`, `IdentitySuspended`, `IdentityReactivated`, `IdentityDeleted`, `TenantDeleted` | — | | `identity.owners` (reactor, order -10) | `TenantCreated` | `CreateIdentity`, `AddOwner` | | `identity.invitations` (reactor, order -10) | `InvitationAccepted` | side effects only | | `identity.invitations` (reactor, order -10) | `InvitationLinkUsed` | side effects only | | `identity.cleanup` (reactor, order -25) | `TenantDeleted` | `DeleteIdentity` | | `identity.follow` (reactor, order -10) | `AccountDeleted` | `DeleteIdentity` | | `identity.follow` (reactor, order -10) | `GroupDeleted` | `RevokeMembership` | --- # Groups A group is a named set of permissions in one tenant: "Customers", "Support", "Accounting". Identities get permissions only through groups (or by owning the tenant), so a tenant's admins manage who may do what without code changes. One aggregate, `group.group`, with a random ID, under its tenant. A group name is unique in its tenant, ignoring case. ## Permissions A permission is a pattern over request types, the same syntax as [grants](../concepts/trust#keys-and-grants), optionally tied to a relation: ```ts CreateGroup.new({ name: "Customers", description: "People who buy in the shop", permissions: [ { pattern: "shop.PlaceOrder" }, // any order { pattern: "shop.CancelOrder", when: "owner" }, // only orders they own { pattern: "shop.GetOrder", when: "owner" }, { pattern: "funds.UpdateVehicle", when: "manager", viaParent: true }, // vehicles of funds they manage ], }); ``` | Field | Means | |---|---| | `pattern` | a type (`shop.PlaceOrder`), every type of a domain (`shop.*`), or everything (`*`) | | `when` | a relation the identity must have to the target aggregate; you declare relations in code, from your events ([Authorization](./authorization#relations)) | | `viaParent` | the relation may hold on the target's parent instead, e.g. the fund a vehicle belongs to | A permission with `when` needs a target: for a command, its aggregate; for a query, the aggregate the query declares it is about ([`static target`](../concepts/queries-and-middleware#designing-queries)). Without a target it never matches. `SetPermissions` replaces the whole list in one event, which keeps the audit readable: each change shows the complete list before and after. ## Owners and platform groups - Every tenant lists a virtual group **`owners`** first: its members are the tenant's owners, it holds `*`, and it cannot be changed or deleted. Owners are managed on the [tenant](./tenancy#owners). - Groups of the **platform tenant** hold their permissions in **every** tenant. That is how the people who run your application (support, billing) get access across customers. The `api` group that `bootstrap()` creates for your API is one of them. ## No escalation Whoever creates a group or changes its permissions may only give permissions they hold themselves. Otherwise anyone allowed to edit groups could give their own group `*`. The same rule applies to granting memberships and to invitations. Only permissions the sender holds for the whole tenant count, without scope or relation; owners and the platform hold all. ## Deleting a group `DeleteGroup` makes its memberships stop counting at once; a reactor then revokes them on each identity. ## Reference ### Types The types the tables below name, from `group/domain.ts`: ```ts interface Permission { pattern: string; when?: string; viaParent?: boolean; } ``` ### Aggregates | Aggregate | Unique keys | Facts | |---|---|---| | `group.group` | `name` | `active` | ### Commands Secret fields leave the vault once the command completes; personal fields stay in the vault until their subject is erased. | Command | Fields | Writes | Who may send it | Refused with | |---|---|---|---|---| | `CreateGroup`
on `group` | `name: string`
`description?: string`
`permissions: Permission[]` | `GroupCreated` | a group permission; only with permissions the sender holds | `exists`, `invalid_name`, `invalid_permissions` | | `DeleteGroup`
on `group` | — | `GroupDeleted` | a group permission | `not_found` | | `SetPermissions`
on `group` | `permissions: Permission[]` | `GroupPermissionsSet` | a group permission; only with permissions the sender holds | `invalid_permissions`, `not_found` | | `UpdateGroup`
on `group` | `name?: string`
`description?: string` | `GroupUpdated` | a group permission | `invalid_name`, `not_found` | ### Events | Event | Fields | Vault | Note | |---|---|---|---| | `GroupCreated` | `name: string`
`description: string`
`permissions: Permission[]` | — | — | | `GroupUpdated` | `name?: string`
`description?: string` | — | — | | `GroupPermissionsSet` | `permissions: Permission[]` | — | The whole list in one event: simpler to read in the audit. | | `GroupDeleted` | — | — | — | ### Queries A query about one aggregate names it (`static target`): the middleware decides on that aggregate as on a command's. | Query | Fields | About | Answer | Who may ask | |---|---|---|---|---| | `GetGroup`
by `group.groups` | `groupId: string` | — | `GroupRow { id, tenant, name, description, permissions }` | a group permission | | `ListGroups`
by `group.groups` | — | — | `(Omit & { tenant?: string })[]` | a group permission | ### Listeners | Listener | On | Submits | |---|---|---| | `group.groups` (read model, order -20) | `GroupCreated`, `GroupUpdated`, `GroupPermissionsSet`, `GroupDeleted`, `TenantDeleted` | — | --- # Invitations An invitation brings a person into a tenant by email: with some groups, or as an owner. It works for people who already have an account and for people who do not, and it is the only way in when the platform's sign-up policy keeps registration closed. An [invitation link](#invitation-links) lets many people in through one link instead. One aggregate, `invitation.invitation`, with a random ID, under the tenant it invites to. It is `open` until it is `accepted`, `declined` or `revoked`; it expires after `INVITATION_DAYS` (7), which is computed, not an event. Each tenant holds at most one open invitation per email. ## The flow ```mermaid sequenceDiagram participant O as admin of the tenant participant K as kernel participant M as invitation.mail participant P as invited person participant A as your API (KitApi) O->>K: CreateInvitation(email, groups) K-->>M: InvitationCreated M->>P: mail with a token P->>A: opens the link A->>K: PeekInvitation (tenant name, email, expired?) P->>A: accepts: signed in, or with a name and password A->>K: AcceptInvitation (+ RegisterAccount for a new person) K-->>K: identity reactor: identity, memberships, owner ``` 1. **Inviting.** `CreateInvitation` with the email and the groups (each optionally scoped to one aggregate). The inviter may only hand out groups they could grant themselves ([no escalation](./authorization#no-escalation)). Owner invitations (`owner: true`) are for owners and the platform. 2. **The mail.** The reactor sends a token of the form `...`, so the token alone tells your API which tenant to accept it in. The token waits in the vault until it is used; `ResendInvitation` replaces it with a new one. 3. **The acceptance page.** Your API calls `peekInvitation(token)` to show the tenant's name and whether the invitation expired. 4. **Accepting**, through `KitApi.acceptInvitation(token, as)`: - **Signed in**: the account's email must be the invited one (else `wrong_email`). - **New person**: they choose a name and password; the account is registered at once and is active, since the token proved the email. Someone whose email already has an account must sign in first (`sign_in_first`), so nobody ends up with an account they cannot use. 5. **Afterwards**, the identity reactor creates the person's identity in the tenant (or keeps the one they have), grants the invitation's groups, and adds them as owner for an owner invitation. The invited person may also **decline** (`DeclineInvitation` with the token). The tenant's admins may **revoke** an open invitation. ## Invitation links An invitation link lets people in without an invitation per email: Ann puts a link on her shop's website, and whoever opens it, signs in and joins gets the link's groups. It is a second aggregate, `invitation.link`: | Command | What it does | |---|---| | `CreateInvitationLink` | a link into `groups` (never as owner), optionally with a `label`, an expiry (`expiresAt`) and a limit (`maxUses`) | | `JoinWithInvitationLink` | the signed-in account joins the tenant with the link's groups; sent by your API | | `RevokeInvitationLink` | the link stops working at once | - **Creating** goes through `KitApi.createInvitationLink(context, tenant, { groups, label?, expiresAt?, maxUses? })`. It generates the token `link_..` and returns it **once**; only its hash is stored, as for API keys. The creator may only hand out groups they could grant themselves ([no escalation](./authorization#no-escalation)). - **Joining** goes through `KitApi.joinWithInvitationLink(claims, token)`. The person must be signed in: unlike an emailed invitation, a link proves no email address, so someone new registers and verifies first, then joins. Each account joins through a link once; a link that expired, is used up or was revoked is refused (`expired_token`, `used_up`, `invalid_token`). - **Afterwards** the identity reactor creates the person's identity in the tenant (or keeps theirs) with the link's groups, exactly as for an accepted invitation. Every join is an event (`InvitationLinkUsed`), so the audit shows who came in through which link. - `peekInvitationLink(token)` gives the join page what it shows: the tenant, the link's label and whether it can still be used. `ListInvitationLinks` lists a tenant's links with how often each was used. ## Tenants created by the platform When the platform creates a tenant for someone (`CreateTenant` with `owner: { email }`), the kit sends that person an owner invitation. Accepting it makes them the tenant's first owner. ## Personal data An invitation keeps its email after it is closed, so the audit shows who was invited and when. `EraseInvitation` removes it, when the person asks or when the tenant is deleted (a reactor does that for every invitation of the tenant). ## Reference ### Types The types the tables below name, from `invitation/domain.ts`: ```ts type InvitationStatus = "none" | "open" | "accepted" | "declined" | "revoked"; interface InvitedGroup { group: string; scope?: { type: string; id: string } | null } type LinkStatus = "none" | "open" | "revoked"; ``` ### Aggregates | Aggregate | Unique keys | Facts | |---|---|---| | `invitation.invitation` | `email` | — | | `invitation.link` | — | — | ### Commands Secret fields leave the vault once the command completes; personal fields stay in the vault until their subject is erased. | Command | Fields | Writes | Who may send it | Refused with | |---|---|---|---|---| | `CreateInvitation`
on `invitation` | `email: string`
`groups?: InvitedGroup[]`
`owner?: boolean` | `InvitationCreated` | a group permission; only with permissions the sender holds | `exists`, `invalid_email`, `invalid_groups`; requires: group.group is active | | `ResendInvitation`
on `invitation` | — | `InvitationResent` | a group permission | `invalid_state` | | `RevokeInvitation`
on `invitation` | — | `InvitationRevoked` | a group permission | `invalid_state` | | `DeclineInvitation`
on `invitation` | `token: string` (secret) | `InvitationDeclined` | anyone (the request carries its proof) | `expired_token`, `invalid_token` | | `AcceptInvitation`
on `invitation` | `token: string` (secret)
`accountId: string`
`email: string`
`newAccount: boolean` | `InvitationAccepted` | the API (its `api` group) | `expired_token`, `invalid_account`, `invalid_token`, `wrong_email` | | `RecordInvitationMailSent`
on `invitation` | `tokenId: string` | `InvitationMailSent` | only the kit's reactors | — | | `EraseInvitation`
on `invitation` | — | `InvitationRevoked`, `InvitationErased` | the platform | `not_found` | | `CreateInvitationLink`
on `link` | `secret: string` (secret)
`groups?: InvitedGroup[]`
`label?: string`
`expiresAt?: number \| null`
`maxUses?: number \| null` | `InvitationLinkCreated` | a group permission; only with permissions the sender holds | `exists`, `invalid_expiry`, `invalid_groups`, `invalid_max_uses`, `invalid_secret`; requires: group.group is active | | `RevokeInvitationLink`
on `link` | — | `InvitationLinkRevoked` | a group permission | `invalid_state` | | `JoinWithInvitationLink`
on `link` | `token: string` (secret)
`accountId: string` | `InvitationLinkUsed` | the API (its `api` group) | `exists`, `expired_token`, `invalid_account`, `invalid_token`, `used_up` | ### Events | Event | Fields | Vault | Note | |---|---|---|---| | `InvitationCreated` | `email: string \| null`
`groups: InvitedGroup[]`
`owner: boolean`
`tokenId: string`
`token: string \| null`
`expiresAt: number` | `email` under `invitation:`; `token` under `token:` | The email belongs to the invitation (its own subject); the token sits in the vault until used. | | `InvitationResent` | `previousTokenId: string`
`tokenId: string`
`token: string \| null`
`expiresAt: number` | `token` under `token:`; erases `token:` | A new token; erases the old one. | | `InvitationRevoked` | `tokenId: string` | erases `usedToken` | — | | `InvitationDeclined` | `tokenId: string` | erases `usedToken` | — | | `InvitationAccepted` | `tokenId: string`
`accountId: string`
`newAccount: boolean`
`groups: InvitedGroup[]`
`owner: boolean` | erases `usedToken` | The identity reactor creates the identity with these groups (and the owner) from it. | | `InvitationMailSent` | `tokenId: string` | — | — | | `InvitationErased` | — | erases `invitation:` | Erases the invited email: when the tenant is deleted, or the person asks. | | `InvitationLinkCreated` | `groups: InvitedGroup[]`
`hash: string`
`label: string`
`expiresAt: number \| null`
`maxUses: number \| null` | — | A reusable invitation: anyone signed in who has its token may join the tenant with these groups, until it is revoked, expires or is used up. Only the token's hash is kept: the API shows the token once. | | `InvitationLinkRevoked` | — | — | — | | `InvitationLinkUsed` | `accountId: string`
`groups: InvitedGroup[]` | — | Someone joined through the link: the identity reactor creates their identity with these groups. | ### Queries A query about one aggregate names it (`static target`): the middleware decides on that aggregate as on a command's. | Query | Fields | About | Answer | Who may ask | |---|---|---|---|---| | `ListInvitations`
by `invitation.invitations` | — | — | `InvitationRow[]` | a group permission | | `PeekInvitation`
by `invitation.invitations` | `token: string` | — | `{ tenant: string; invitationId: string; email: string \| null; owner: boolean; expired: boolean }` | the API (its `api` group) | | `ListInvitationLinks`
by `invitation.invitations` | — | — | `LinkRow[]` | a group permission | | `PeekInvitationLink`
by `invitation.invitations` | `token: string` | — | `{ tenant: string; linkId: string; label: string; usable: boolean }` | the API (its `api` group) | ### Listeners | Listener | On | Submits | |---|---|---| | `invitation.invitations` (read model, order -20) | `InvitationCreated`, `InvitationResent`, `InvitationRevoked`, `InvitationDeclined`, `InvitationAccepted`, `InvitationErased`, `InvitationLinkCreated`, `InvitationLinkRevoked`, `InvitationLinkUsed`, `TenantDeleted` | — | | `invitation.mail` (reactor, order -10) | `InvitationCreated` | side effects only | | `invitation.mail` (reactor, order -10) | `InvitationResent` | side effects only | | `invitation.owners` (reactor, order -10) | `TenantCreated` | `CreateInvitation` | | `invitation.cleanup` (reactor, order -25) | `TenantDeleted` | `EraseInvitation` | --- # Authorization The kit decides every request in one place: a [middleware](../concepts/queries-and-middleware#middleware) named `authorization`, which runs before every command, query, file and cache request of every domain, yours included. It decides from what it learned from events (tenants, accounts, sessions, identities, groups, and the relations you declare), so a decision needs no database call and no request to another service. It runs at order -30, before any middleware of yours (yours run at 0 or later). While it is away the kernel refuses every request with `UNAVAILABLE`: it fails closed. ## The decision Each request passes these steps in order. The first one that decides, decides. | Step | Allows | Refuses with | |---|---|---| | 1. Privileged request | requests marked `bypassAuthorization`, which only keys granted `privileged` may send: the kit's reactors and `bootstrap()` | — | | 2. Public types | `SignIn`, `VerifyEmail`, `ResetPassword` and others that carry their own proof (a password, a token) | `sign_up_closed`: `RegisterAccount` while registration is by invitation only | | 3. Who is asking | — | `unauthenticated`: no active session of an active account, and no valid API key. An account with a second factor on may only sign out until the session proved it | | 4. Signed in is enough | e.g. `ListMyIdentities`; `CreateTenant` for oneself when the sign-up policy allows it | — | | 5. The tenant | — | `forbidden`: not a member; `tenant_suspended`: the tenant is suspended | | 6. The identity | — | `forbidden`: no active identity of this account in the tenant; `mfa_required`: the tenant requires a second factor | | 7. Owners | everything in their tenant | `sso_required`: the tenant enforces SSO and the session did not use it (owners exempt) | | 8. Memberships | an active, unexpired membership of an active group with a permission that matches the type, in scope, with its relation holding | `forbidden` | | 9. No escalation | — | `forbidden`: granting a group, inviting into one, or setting permissions beyond what the sender holds | A refusal reaches the client as a rejection with the code. A refused command is never stored, so a flood of refused requests cannot fill the log. ## Relations A relation says who stands in which relation to an aggregate: the `owner` of an order, the `manager` of a fund. You declare relations next to your aggregates, **from your events**; the middleware keeps them up to date as the events arrive: ```ts [TypeScript] import { parent, relation } from "./kit/src/kit.ts"; relation("shop.order", "owner") .set("shop.OrderPlaced", (e) => e.data.customer) // an identity ID, or a list .unset("shop.OrderTransferred", (e) => e.data.from) .set("shop.OrderTransferred", (e) => e.data.to); // Permissions with `viaParent` look at the fund a vehicle belongs to. parent("funds.vehicle").from("funds.VehicleCreated", (e) => ({ type: "funds.fund", id: e.data.fund })); ``` ```python [Python] from kit import parent, relation (relation("shop.order", "owner") .set("shop.OrderPlaced", lambda e: e.data["customer"]) .unset("shop.OrderTransferred", lambda e: e.data["from"]) .set("shop.OrderTransferred", lambda e: e.data["to"])) parent("funds.vehicle").from_("funds.VehicleCreated", lambda e: {"type": "funds.fund", "id": e.data["fund"]}) ``` ```rust [Rust] use kit::authorization::{ids, parent, relation}; relation("shop.order", "owner") .set("shop.OrderPlaced", |e| ids(&e.data["customer"])) .unset("shop.OrderTransferred", |e| ids(&e.data["from"])) .set("shop.OrderTransferred", |e| ids(&e.data["to"])); ``` For queries, the middleware needs to know which aggregate a query is about. Declare it on the query, once: `static target = { aggregateType: "order", field: "id" }` on a query class, or `shopDomain.query("GetOrder").target("order", "id")` for a query given by name. Then `{ pattern: "shop.GetOrder", when: "owner" }` lets each customer read only their own orders. Declare relations **before** `kitDomains()`, which hands them to the middleware. The names `self` and `owners` are built in. A group permission then uses the relation with `when: "owner"` ([Groups](./group#permissions)). A membership scoped to one aggregate counts as a relation set by hand. ## Lists: `visibility` The middleware decides single requests. A query that answers a **list** (all orders, all invoices) is allowed or refused as a whole, so its read model must filter the rows itself. `visibility(readModel)` teaches your read model what the middleware knows and returns a function that decides the same way per row: ```ts import { visibility } from "./kit/src/kit.ts"; const orders = shop.readModel("shop.orders"); const visible = visibility(orders); orders.query(ListOrders, (_d, qry, ctx) => rows.filter((o) => visible(qry.tenant, ctx.request, "shop.GetOrder", { type: "shop.order", id: o.id }))); ``` Each row shows only if the asker could `GetOrder` it. The knowledge is kept in the read model's own store, so it resumes after a restart. ## Why was it refused: `Explain` `group.Explain` answers whether an identity may send a type, for an aggregate if you give one, with every step of the decision: ```ts const answer = await kernable.query(Explain.new({ identityId, type: "shop.CancelOrder", aggregateType: "shop.order", aggregateId: "o-17", }), { tenant, context }); // { allowed: false, code: "forbidden", steps: [one line per step it passed, and the one that refused] } ``` Anyone may explain their own requests; explaining someone else's needs the `group.Explain` permission. Support screens and admin tools use it to show *why*. ## Your own rules Rules that permissions and relations cannot express ("orders above 10,000 need a second approver") belong in a middleware of your own. It runs after the kit's, so it sees only requests that passed, with the identity already checked. ## Where it keeps its knowledge The middleware keeps what it learned in a store (`useAuthorizationStore("sqlite:authorization.db")` by default). After a restart it continues from where it stopped instead of reading the whole log again, which matters, because it refuses every request until it has caught up. ## Settings The levels of the kit's own types (who may send `SignIn`, `SuspendTenant` and so on) are data, in `authorization/levels`. The reference tables of each domain show them under "Who may send it". Types not listed there, and all of yours, need a group permission. The decision itself is in `authorization/policy`, one function you can read top to bottom. --- # Your API: KitApi Your HTTP API is the one process that sees passwords, tokens and answers from identity providers. `KitApi` is the part of the kit that runs there: it turns "email and password" into signed commands, issues and checks tokens, runs passkey and OIDC exchanges, and turns a token or API key into the request context your own commands carry. It is framework-neutral: plain methods you call from Express, FastAPI, axum or anything else. `kit/example/server.ts` (`server.py`, `examples/server.rs`) wires it into a small server on the standard library; copy its routes into your framework. ## Setting it up ```ts import { KitApi } from "./kit/src/api/api.ts"; import { signingKeysFrom } from "./kit/src/api/tokens.ts"; const api = new KitApi({ kernable, // a client with the API's key keys: signingKeysFrom("token-signing.pem"), // EdDSA key that signs access tokens service: { identityId: "api", keyId }, // from bootstrap(): the API's own identity providers: { // optional: social logins google: { issuer: "https://accounts.google.com", clientId, clientSecret }, }, redirectUri: "https://example.com/oidc/callback", // where providers send the browser back }); ``` In Python the same is `KitApi(kernable, signing_keys_from(...), service)`; it blocks like the SDK's client, and `AsyncKitApi(KitApi(...))` offers the same methods for asyncio. In Rust, `KitApi::new(...)` is async. Every method has the same name in snake_case. The API's own identity holds a few permissions the kit needs before anyone is signed in, such as finding an account by email or a tenant by slug. `bootstrap()` creates it with the `api` group; see `API_PERMISSIONS` in `bootstrap`. ## Per request ```ts const claims = api.check(accessToken); // the token's claims, or null; no kernel call const context = await api.contextFor(claims, tenantId); // senderAccount, senderSession, senderTenant, senderIdentity await kernable.submit(PlaceOrder.new({ … }), { aggregateId, tenant: tenantId, context }); ``` - `check` verifies the access token's signature and expiry locally. - `contextFor` adds the person's identity in the tenant; without a tenant, the context is only the signed-in account (enough for `authenticated` requests such as listing one's tenants). - For a service, `checkApiKey(key)` returns the context directly (the key's identity and tenant). The context goes with the command; the [middleware](./authorization) decides on it. Your API does not decide who may do what. ## Methods | Task | Method | Returns | |---|---|---| | **Registration** | `register({ email, name, password? })` | `{ accountId }`, the same whether or not the address was taken | | | `verifyEmail(token)` | whether it worked | | **Password sign-in** | `signIn({ email, password, ip?, userAgent? })` | access and refresh token, session ID, and `secondFactor: true` if the account must prove one now; or a reason | | | `refresh(refreshToken, ip?)` | a new access token and the next refresh token | | | `signOut(claims)` | ends the session | | **Second factor** | `proveSecondFactor(claims, code)` | a new access token with the factor, for a TOTP or recovery code | | | `startTotp(claims, issuer, label)` | the secret and the `otpauth:` URI to show as a QR code | | | `confirmTotp(claims, code)` | the recovery codes, shown this once | | **Passkeys** | `passkeyRegistrationOptions(claims, userName)`, then `addPasskey(claims, response, label?)` | adds a passkey the browser created | | | `passkeySignInOptions()`, then `signInWithPasskey(response, ip?)` | signs in without a password | | | `provePasskey(claims, response)` | a passkey as the second factor of a password session | | **OAuth/OIDC** | `oidcStart({ provider } \| { email }, link?)` | the provider's URL to redirect to; `{ email }` finds the tenant SSO of the email's domain; with `link`, links the login to the signed-in account | | | `oidcFinish(callbackUrl, ip?)` | signs in, registers or links (see [Accounts](./account#other-ways-to-sign-in)) | | | `verifySsoDomain(tenant, ssoId, domain)` | looks up the `_kernable-sso.` TXT record and verifies the domain | | **Account changes** | `changePassword(claims, current, next)` | ends every other session | | | `requestPasswordReset(email)`, `resetPassword(token, next)` | the reset by mail; the request answers the same for unknown addresses | | | `requestEmailChange(claims, email)`, `confirmEmailChange(token)` | the change, confirmed from the new address | | **Invitations** | `peekInvitation(token)` | tenant name, email, whether it expired | | | `acceptInvitation(token, { claims } \| { name, password })` | as the signed-in account, or as a new person ([Invitations](./invitation#the-flow)) | | **Invitation links** | `createInvitationLink(context, tenant, { groups, label?, expiresAt?, maxUses? })` | the link's token, shown this once | | | `peekInvitationLink(token)` | tenant, label, whether it can be used | | | `joinWithInvitationLink(claims, token)` | the signed-in account joins with the link's groups ([Invitation links](./invitation#invitation-links)) | | **Services** | `createApiKey(context, tenant, identityId, label?, expiresAt?)` | the key, shown this once | | | `checkApiKey(key)` | the request context of the key, or null | ## Security notes - **Nothing reveals whether an account exists.** Sign-in answers `wrong_password` for unknown addresses, and registration and password reset answer the same either way. - **Tokens are short and narrow.** Access tokens last 15 minutes and are checked without the kernel; the middleware still refuses an ended session at once. Refresh tokens rotate, and a reused one ends its session. - **Passkey challenges and OIDC state** live a few minutes in the kernable cache, so any instance of your API can finish what another started. That is why the API's key needs `cache` on `account`. - **Secrets never stay in the log.** Passwords, tokens and codes travel as secret command fields and leave the vault once the command completes; hashes and personal data stay in the vault and are erased with the person. --- # Recipes Common changes to the kit, each in a few steps. The examples are TypeScript; the Python and Rust kits have the same files and names in their style. ## Open registration, or invitations only The platform's sign-up policy decides who may register and who may create tenants. Set it in `bootstrap()`, or change it later as the platform: ```ts await bootstrap(kernable, { signUp: { accounts: "open", tenants: "active" } }); // a typical SaaS await bootstrap(kernable, { signUp: { accounts: "off", tenants: "off" } }); // invitation only ``` | You want | `accounts` | `tenants` | |---|---|---| | anyone signs up and creates their own workspace | `open` | `active` | | anyone signs up; new workspaces wait for your approval (`ActivateTenant`) | `open` | `pending` | | you create tenants; people join by invitation | `off` | `off` | Later: `ChangeSignUpPolicy.new({ accounts, tenants })` on the platform tenant. ## Let people join with a link An owner creates a link with the groups newcomers get, and puts it on a website or in a chat: ```ts const link = await api.createInvitationLink(ownerContext, tenant, { groups: [{ group: customersId }], label: "website", maxUses: 100 }); // link.token, shown once: https://example.com/join?token= ``` Your join page calls `api.peekInvitationLink(token)` to show the tenant and whether the link still works, asks the person to sign in (or sign up and verify), then calls `api.joinWithInvitationLink(claims, token)`. Part 9 of the [shop tutorial](../tutorial/shop/9-sign-in) does exactly this. ## Let support staff into every tenant Groups of the [platform tenant](./tenancy#the-platform-tenant) hold their permissions in every tenant. Create a group there and invite your staff into it: ```ts await kernable.submit(CreateGroup.new({ name: "Support", description: "Reads customers' data to help them", permissions: [{ pattern: "shop.GetOrder" }, { pattern: "identity.GetIdentity" }, { pattern: "group.Explain" }], }), { aggregateId: randomUUID(), tenant: PLATFORM_TENANT, context }); await kernable.submit(CreateInvitation.new({ email: "ann@example.com", groups: [{ group: supportGroupId }] }), { aggregateId: randomUUID(), tenant: PLATFORM_TENANT, context }); ``` Everything they do is in the log with their identity, in the platform tenant, as the sender. ## Require a second factor in a tenant An owner of the tenant sends `RequireMfa.new({ required: true })` on the tenant. From then on, sessions without TOTP or a passkey get `mfa_required` there. Your UI should catch that code and lead the person to set up TOTP ([`startTotp`, `confirmTotp`](./api#methods)) or a passkey. ## Single sign-on for a customer 1. An owner of the customer's tenant sends `CreateSso` with the identity provider's issuer, client ID and client secret (the secret goes to the vault). 2. They add their email domain: `AddSsoDomain({ domain: "acme.com" })`. The event carries the value for the DNS record. 3. They publish a TXT record `_kernable-sso.acme.com` with that value; your API calls `api.verifySsoDomain(tenant, ssoId, "acme.com")`. 4. Your sign-in page asks for the email first and, for a verified domain, sends the person to `api.oidcStart({ email })`. 5. Optionally `EnforceSso({ enforced: true })`: in this tenant, only SSO sessions count (owners exempt). ## A service that calls your API ```ts const created = await kernable.submit(CreateServiceIdentity.new({ displayName: "Billing sync" }), { aggregateId: serviceId, tenant, context }); await kernable.submit(GrantMembership.new({ membershipId: randomUUID(), group: billingGroupId }), { aggregateId: serviceId, tenant, context }); const key = await api.createApiKey(context, tenant, serviceId, "billing sync"); // Show key.key once. The service sends `Authorization: ApiKey `; your API calls api.checkApiKey(). ``` Revoke it with `RevokeApiKey({ keyId })`; the middleware refuses it at once. ## Show only what someone may see in a list Use [`visibility`](./authorization#lists-visibility) in the read model that answers the list, with the same type its single-item query has (`shop.GetOrder`), so lists and detail pages agree. ## Change a lifetime or a limit Edit `shared/constants`: lockout, token and session lifetimes, invitation validity, reserved slugs. If a test case in `kit/fixtures` covers the old value, change the case too. That is deliberate: the cases are the contract of the kit as you run it. ## Add a field Say identities should have a phone number. 1. **Event**: add `phone!: string | null;` to `IdentityCreated` and `ProfileUpdated` in `identity/events.ts`, and add `"phone"` to their `personal` fields, so it goes to the vault and is erased with the identity. 2. **Command**: add `phone?: string;` to `UpdateProfile` in `identity/commands.ts`, and to its `personal` fields; validate it in the handler and pass it on to the event. 3. **State and rows**: apply it in `identity/aggregates.ts` if a rule needs it, and add it to the row in `identity/readmodels.ts` so `GetIdentity` returns it. 4. **Cases**: add `"phone": null` to the expected events in `kit/fixtures/identity.json`, and a case for a valid and an invalid number. Events already in your log have no `phone`; read it as missing (`?? null`). Once kernable 1.0 is out, use [event versions](../how-to/event-versions) for changes to stored events. ## Remove a domain you do not need Say you never invite anyone. Delete `invitation/` and remove it from `kitDomains()`, then let the compiler (or your tests) show what still refers to it: the identity reactor that turns accepted invitations into identities, `acceptInvitation` and `peekInvitation` in `KitApi`, the owner invitation for tenants the platform creates, and `fixtures/invitation.json`. Decide for each whether you need a replacement. For example, tenants created by the platform then need their first owner some other way, such as `CreateTenant` with `owner: { accountId }`. Removing a domain from a running application also means its listeners stop: [Remove a listener or aggregate](../how-to/remove-listener) explains how the kernel stops waiting for them. --- # Reference Look things up here; the [concepts](../concepts/) explain why they are so. Most pages are generated from the sources and checked in every build, so they match the code they ship with: the flags, the CLI, the protocol, the metrics and the three SDKs. [Signatures](./signing) is included from `proto/SIGNING.md`. [Errors](./errors) is written by hand, and the build fails if a rejection code in the sources is missing from it. | Page | What it covers | |---|---| | [kernable start](./configuration) | Every flag and environment variable of the kernel, with defaults, by topic: trust, stores, network, timeouts; `--dev` | | [CLI](./cli) | The `kernable` command: run the kernel; keys, grants, revocation and certificates; apply a membership and list services, workers, listeners and schemas; verify and checkpoint a store; remove registrations, rekey the vault; submit a command | | [TypeScript SDK](./sdk-typescript) | `@kernable/sdk` | | [Python SDK](./sdk-python) | `kernable` for Python | | [Rust SDK](./sdk-rust) | `kernable` for Rust | | [Protocol](./protocol) | The gRPC services and messages (`kernable.v1`), for writing an SDK | | [Signatures](./signing) | Signed byte layouts, the hash chain, vault references, test vectors | | [Errors](./errors) | Rejection codes and gRPC statuses, and what a client should do about each | | [Metrics and health](./metrics) | Prometheus metrics, `/health` and `/ready` | The kernel's UI in a browser (`--ui-listen`) has no reference page: [Monitor kernable](../how-to/monitor#the-ui) describes it. --- # kernable start The kernel takes its settings as flags of `kernable start`. Every flag has an environment variable. A flag wins over its variable, and the variable over the default. Write a flag as `--flag value` or `--flag=value`; given twice, the last one wins, except `--root`, which adds a root each time. Relative paths resolve against the working directory. `kernable start --help` lists the flags. An unknown flag, a missing value or a malformed one ends `kernable start` with exit code 2 before the kernel starts. ```sh kernable start --root root.pub --store sqlite:kernable.db --listen 127.0.0.1:7233 ``` The kernel holds no keys of services and no permissions: it trusts the root keys, and a root key says with a grant what each key may do ([Trust](../concepts/trust), [Manage keys](../how-to/keys)). SIGHUP (Unix) reads the kernel key file again; every other change needs a restart. **`--dev`** is for trying kernable out on one machine. It creates whatever is missing of `root.key`, `kernel.key`, `vault.key`, `app.key` and `app.grant` in the working directory, with a grant that lets `app.key` do everything. Unless you set them by flag or variable, it then uses `--root root.pub`, `--vault-key vault.key` and `--ui-listen 127.0.0.1:7234`. Every other flag keeps its default. In production, start with `--root` and grant each service only what it needs. **Logs** go to stderr. `KERNABLE_LOG` sets the filter (default `info`) and `KERNABLE_LOG_FORMAT=json` writes JSON ([Monitor kernable](../how-to/monitor#logs)). ## Trust Which keys the kernel trusts, and its own key. | Flag | Variable | Default | Description | |---|---|---|---| | `--root` | `KERNABLE_ROOT` | **required** | A root key the kernel trusts: its key ID (hex) or the path of its `.pub` file. Only a root key can sign grants and revocations. Repeat the flag for several roots, for example while you replace one; the variable takes them comma-separated. At least one is required; `--dev` uses `root.pub` if none is given. | | `--kernel-key` | `KERNABLE_KERNEL_KEY` | `kernel.key` | The kernel's own signing key, written by `kernable keys new kernel`. It signs checkpoints and stored revocations. The file must exist. On Unix, SIGHUP reads it again, so a new key signs from then on without a restart. | ## Stores Where the kernel keeps what. A store URL names a backend: `sqlite:` (or a bare path), `file:` or `memory:`; not every store takes every backend. `memory:` keeps nothing across a restart and is for tests. What to back up: [Back up and restore](../how-to/backup). | Flag | Variable | Default | Description | |---|---|---|---| | `--store` | `KERNABLE_STORE` | `sqlite:kernable.db` | The event store. It also holds the command queue and results, checkpoints, revocations and registrations. `sqlite:` or a bare path: SQLite. `file:`: an append-only log file. `memory:`: nothing survives the process, for tests. | | `--durability` | `KERNABLE_DURABILITY` | `full` | When a write to the event store counts as done. `full`: on the disk (fsync, a full flush on macOS) before a client, listener or worker hears of it, so a committed command survives a power loss or an OS crash. Commits waiting at the same time share one flush (group commit), so throughput holds up under concurrent load, while a single client waits about one flush per command. `normal`: handed to the OS, which survives a crash of the kernel but not of the machine, and is faster; `--dev` uses it unless given. The vault and the data service always write to the disk. | | `--snapshots` | `KERNABLE_SNAPSHOTS` | — | The snapshot store, a cache of aggregate states that workers rebuild from the events. `sqlite:` or a bare path: SQLite, kept across restarts. `memory:`: lost on a restart, so the first command per aggregate reads all its events again. Not `file:`. Default: SQLite next to the event store, `-snapshots.db` (`kernable-snapshots.db` for `kernable.db`), or `memory:` for a `memory:` store. | | `--vault-key` | `KERNABLE_VAULT_KEY` | — | The key that encrypts the vault: a file with 32 random bytes, hex encoded (`kernable keys new vault` writes one). Without it the kernel has no vault: commands and events with vault values are refused. `--dev` uses `vault.key` if none is given. | | `--vault-store` | `KERNABLE_VAULT_STORE` | `sqlite:vault.db` | Where the vault keeps personal and secret values, encrypted with `--vault-key`. `sqlite:` or a bare path: SQLite. `memory:`: every value is lost on a restart, for tests. Not `file:`. Ignored without `--vault-key`. | | `--data-store` | `KERNABLE_DATA_STORE` | `file:data` | Where the data service keeps files. `file:`: one file per stored file, the directory is created with the first one. `memory:`: lost on a restart. | | `--cache-store` | `KERNABLE_CACHE_STORE` | `memory:` | Where the cache service keeps values. Only `memory:` exists: values last until they expire, are deleted, or the kernel restarts. | ## Network Where the kernel listens. Plain TCP only on a loopback address; anything else needs mutual TLS ([Set up TLS](../how-to/tls)). The metrics and UI addresses serve plain HTTP with no login ([Monitor kernable](../how-to/monitor)). | Flag | Variable | Default | Description | |---|---|---|---| | `--listen` | `KERNABLE_LISTEN` | `127.0.0.1:7233` | The gRPC address for every service and client: `host:port`, or `unix:` on Unix, whose owner and group may connect (mode 0660). Plain TCP is accepted only on a loopback address (`127.0.0.1`, `::1`, `localhost`); any other address needs mutual TLS (`--tls-cert`, `--tls-key`, `--tls-client-ca`), or the kernel does not start. | | `--tls-cert` | `KERNABLE_KERNEL_TLS_CERT` | — | Mutual TLS on a TCP `--listen`: the kernel's certificate (PEM). `kernable certs new` writes development certificates. Give all three TLS flags or none; TLS with a `unix:` address is an error. | | `--tls-key` | `KERNABLE_KERNEL_TLS_KEY` | — | Mutual TLS: the private key of `--tls-cert` (PEM). | | `--tls-client-ca` | `KERNABLE_TLS_CLIENT_CA` | — | Mutual TLS: the CA certificate (PEM) that must have signed every client's certificate. A client without one is refused. | | `--metrics-listen` | `KERNABLE_METRICS_LISTEN` | — | A TCP `host:port` for plain HTTP with no login: Prometheus metrics at `GET /metrics`, `GET /health` and `GET /ready`. Bind it to a private address. Off when empty. If the address cannot be bound, the kernel stops. | | `--ui-listen` | `KERNABLE_UI_LISTEN` | — | A TCP `host:port` for the kernel's UI in a browser: an overview of health and activity, the business processes and their cases, operations (services, lag, parked events, rejections, standby, store), and the domains with their commands, events and readers, the log and every aggregate's history. It has no login, so the address must be loopback, or the kernel does not start. If the address is taken, the kernel runs on without the UI and logs `the UI is off`. Off when empty; `--dev` sets `127.0.0.1:7234` unless it is given. | ## Timeouts and intervals A timeout ends the wait, not the work: what happens then is in each row and in [Failures and recovery](../concepts/failures). | Flag | Variable | Default | Description | |---|---|---|---| | `--lease-ttl-ms` | `KERNABLE_LEASE_TTL_MS` | `10000` | How long a worker holds a command, in milliseconds. If its result has not arrived by then, the command goes to another worker and a late result is ignored. A worker that disconnects gives its commands back at once. Set it above your slowest command handler. | | `--checkpoint-interval-ms` | `KERNABLE_CHECKPOINT_INTERVAL_MS` | `5000` | How often, in milliseconds, the kernel signs a checkpoint of the hash chain with `--kernel-key`, if new events arrived. Events after the last checkpoint are not yet covered by a signature (`kernable verify` counts them). | | `--submit-timeout-ms` | `KERNABLE_SUBMIT_TIMEOUT_MS` | `30000` | How long a submit waits for its command to be committed or rejected, in milliseconds. After that the client gets `DEADLINE_EXCEEDED`, but the command stays queued and still runs: resubmit it with the same command ID to get the result. | | `--settle-timeout-ms` | `KERNABLE_SETTLE_TIMEOUT_MS` | `5000` | How long a settling submit (the default) waits after the commit for the listeners that count to process the events, in milliseconds. After that the client gets the committed outcome marked as not settled (`NotSettled` in the SDKs). | | `--query-timeout-ms` | `KERNABLE_QUERY_TIMEOUT_MS` | `5000` | How long a query waits, in milliseconds, for a read model that answers it to catch up and answer. After that the client gets `DEADLINE_EXCEEDED`. | | `--middleware-timeout-ms` | `KERNABLE_MIDDLEWARE_TIMEOUT_MS` | `5000` | How long a request waits for each middleware in its chain to catch up and give its verdict, in milliseconds. After that the request is refused with `DEADLINE_EXCEEDED`: middleware fails closed. | | `--drain-timeout-ms` | `KERNABLE_DRAIN_TIMEOUT_MS` | `10000` | On SIGTERM or SIGINT, the kernel refuses new submits and queries and waits this long, in milliseconds, for running ones to finish before it exits. Commands still unfinished run again after a restart. | ## Limits What the kernel takes on at most, so that no client, service or connection can exhaust its memory or capacity. Beyond a limit it refuses with `RESOURCE_EXHAUSTED` (a connection beyond `--max-connections` is closed) and counts the refusal in `kernable_refused_total` ([Metrics](./metrics)). Every limit is at least 1. | Flag | Variable | Default | Description | |---|---|---|---| | `--max-connections` | `KERNABLE_MAX_CONNECTIONS` | `1024` | Connections to the gRPC address at once. A connection beyond it is closed right away and counted in `kernable_refused_total{reason="connections"}`. Keep it below the process's limit of open files. | | `--max-streams-per-connection` | `KERNABLE_MAX_STREAMS_PER_CONNECTION` | `256` | Requests and streams at once on one connection (HTTP/2 concurrent streams). Further ones wait in the client until one ends. | | `--max-message-bytes` | `KERNABLE_MAX_MESSAGE_BYTES` | `4194304` | The largest gRPC message the kernel receives or sends, in bytes; the SDKs' default is the same 4 MiB. The data service takes files up to 16 MiB regardless. A command whose snapshot and the events after it do not fit one message reaches the worker in parts; only a command with its snapshot, or a single event, that does not fit one message is rejected with `too_large`. Listener batches are cut to fit. | | `--handshake-timeout-ms` | `KERNABLE_HANDSHAKE_TIMEOUT_MS` | `10000` | How long a worker or listener stream may take to send its first message and to answer the challenge, in milliseconds. A stream that is slower ends with `DEADLINE_EXCEEDED`. | | `--max-queued-commands` | `KERNABLE_MAX_QUEUED_COMMANDS` | `100000` | Commands in the queue at once, waiting or running. A new command beyond it is refused with `RESOURCE_EXHAUSTED` and not stored; resubmitting one that is already stored is always answered. | | `--max-queued-per-aggregate` | `KERNABLE_MAX_QUEUED_PER_AGGREGATE` | `1000` | Commands in the queue at once for one aggregate. A new command beyond it is refused with `RESOURCE_EXHAUSTED`, so one busy aggregate cannot fill the queue. | | `--max-queued-bytes` | `KERNABLE_MAX_QUEUED_BYTES` | `536870912` | Bytes of commands in the queue at once (the queue lives in the kernel's memory). A new command beyond it is refused with `RESOURCE_EXHAUSTED`. | | `--max-requests-per-key` | `KERNABLE_MAX_REQUESTS_PER_KEY` | `1000` | Submits, queries, file and cache requests of one key in progress at once. One beyond it is refused with `RESOURCE_EXHAUSTED`, so one service cannot take all the kernel's capacity. | | `--max-subscriptions-per-key` | `KERNABLE_MAX_SUBSCRIPTIONS_PER_KEY` | `16` | Open event subscriptions (`EventService.Subscribe`) of one key at once. One beyond it is refused with `RESOURCE_EXHAUSTED`. | | `--max-listeners-per-key` | `KERNABLE_MAX_LISTENERS_PER_KEY` | `64` | Listeners one key may register (connected or not) or run live at once. A new name beyond it is refused with `RESOURCE_EXHAUSTED`; registered ones always reconnect. A registered listener counts for settling and readiness, so this bounds how many a key can make everyone wait for. | | `--max-cache-bytes` | `KERNABLE_MAX_CACHE_BYTES` | `268435456` | Bytes the cache service keeps (values and keys). When a value would exceed it, the oldest values are dropped first, as a cache may; a single value larger than the limit is refused with `RESOURCE_EXHAUSTED`. | | `--max-data-bytes` | `KERNABLE_MAX_DATA_BYTES` | `16777216` | Bytes of one file of the data service, at most 134217728 (128 MiB). A larger put is refused with `INVALID_ARGUMENT`. A file travels whole in one message, so the kernel and the SDK hold it in memory while it is put or read. | ## Other | Flag | Variable | Default | Description | |---|---|---|---| | `--replication` | `KERNABLE_REPLICATION` | `off` | `sync`: a standby (`--follow`) holds every write before anyone is told of it. Commits, revocations, grants and memberships wait until the standby has stored them; without a standby in sync they wait, and `/ready` answers 503. Without a witness, until a standby has caught up once (a note beside the store keeps it), the kernel serves alone and the standby catches up meanwhile; only a standby told it caught up may be promoted. `off`: no standby. | | `--backup` | `KERNABLE_BACKUP` | `off` | `on`: a backup follower (`kernable backup`, a key granted `backup`) may follow the kernel and write its records to object storage; the kernel keeps its record log for it, as for a standby. `off`: no backup follower. | | `--partition` | `KERNABLE_PARTITION` | — | This kernel's partition, a name in the partition map: it serves the tenants the map assigns to it and refuses the others with `FAILED_PRECONDITION` `wrong_partition`, and none before it has a map. Without it, the kernel serves every tenant. | | `--control` | `KERNABLE_CONTROL` | — | With `--partition`: the control partition's addresses, comma-separated (both kernels of a pair). This kernel takes the partition map, grants, revocations and memberships from there every second, with `--node-key` (granted `replicate`). Without it, this kernel is the control partition: maps are applied here (`kernable partitions apply`). | | `--witness` | `KERNABLE_WITNESS` | — | Run as one kernel of a pair whose roles a witness (`kernable witness`) decides, at this address: the primary of the newest term serves with `--replication sync`, the other follows it, and takes its place on its own once the primary's lease runs out. Needs `--node-key` and `--primary-kernel-key`; not with `--follow`. The connection uses `KERNABLE_TLS_*`, as the SDKs do. | | `--node-key` | `KERNABLE_NODE_KEY` | `node.key` | With `--witness` or `--control`: this kernel's key towards the witness, the other kernel and the control partition, whose grant (`.grant` next to it) allows `replicate`. | | `--advertise` | `KERNABLE_ADVERTISE` | — | With `--witness`: the address the other kernel reaches this one at. Default: `--listen`. | | `--standby-timeout-ms` | `KERNABLE_STANDBY_TIMEOUT_MS` | `5000` | With `--witness`: how long the primary's writes wait for a standby that is away or behind before the primary tells the witness and commits alone. A standby that missed writes is never made the primary. | | `--follow` | `KERNABLE_FOLLOW` | — | Run as the standby of the primary at this address: apply its records, serve nothing else (every request is refused with `UNAVAILABLE`, `not_primary`) until `kernable promote` makes this kernel the primary. The connection uses `KERNABLE_TLS_CA`, `KERNABLE_TLS_CERT` and `KERNABLE_TLS_KEY`, as the SDKs do. | | `--follow-key` | `KERNABLE_FOLLOW_KEY` | `standby.key` | With `--follow`: the standby's key, whose grant (`.grant` next to it) allows `replicate`. | | `--primary-kernel-key` | `KERNABLE_PRIMARY_KERNEL_KEY` | — | With `--follow`: the primary's kernel key, as a key ID or a `.pub` file; the standby checks the primary's checkpoints with it. Repeat the flag for a key the primary had before; the variable takes them comma-separated. | | `--take-over-ms` | `KERNABLE_TAKE_OVER_MS` | `0` | Start while another kernel still holds the stores, and take them over (a kernel update on the same machine): wait at most this long, in milliseconds, for it to let them go, then serve on the same address. The other kernel lets go when it gets SIGTERM, once its running requests finished. `kernable node` does this when its kernel's binary changes. 0: refuse stores another kernel holds. | | `--schedule-max-days` | `KERNABLE_SCHEDULE_MAX_DAYS` | `366` | How far ahead, in days, a scheduled command (`Command.deliver_at`) may be due. One due later is refused with `INVALID_ARGUMENT`. Scheduled commands wait in the kernel's memory until due and count towards `--max-queued-commands`, `--max-queued-per-aggregate` and `--max-queued-bytes`. | | `--key-command` | `KERNABLE_KEY_COMMAND` | — | A program and its arguments, separated by spaces, that unwraps a key: the kernel runs it with a key file's path as the last argument and reads the key from its output (64 hex characters, as in a key file), for the kernel key, the vault key and the node keys, at start and on SIGHUP. The key files then hold wrapped keys (age, a KMS, systemd-creds). Empty: the files hold the keys. | | `--max-store-bytes` | `KERNABLE_MAX_STORE_BYTES` | `0` | Bytes the event store, the vault and the files may take together, measured every 10 s (0: no limit). At the limit, new commands and file puts are refused with `RESOURCE_EXHAUSTED` `quota_storage`; queries, listeners, file deletes and the commands of `--quota-exempt` go on. At 80 % the kernel logs a warning. | | `--quota-exempt` | `KERNABLE_QUOTA_EXEMPT` | — | Commands `--max-store-bytes` never refuses, comma-separated (`identity.ErasePerson`, `identity.*`): list the commands that erase personal data, so an erasure request is always possible. | | `--max-requests-per-second-per-key` | `KERNABLE_MAX_REQUESTS_PER_SECOND_PER_KEY` | `0` | Submits, queries, file and cache requests one key may start per second, with a burst of one second's worth (0: no limit). More are refused with `RESOURCE_EXHAUSTED` `rate`, to retry a moment later. | --- # kernable (CLI) The `kernable` command runs the kernel and does everything around it: it creates keys, grants and development certificates, revokes keys, applies a service's membership, lists what is registered, checks stores, and submits single commands for scripting. `kernable` without arguments prints this page as text. The flags of `kernable start` are in [kernable start](./configuration). Commands that take an `` talk to a running kernel. Commands that take a `` open the store file directly: the ones that only read it work while the kernel runs, the ones that change it need the kernel stopped. ## Run the kernel ### `kernable start` ```text kernable start --root [ ]... kernable start --dev [ ]... ``` Runs the kernel in the foreground, trusting grants signed by the root keys; the flags are listed by `kernable start --help`. On Unix, SIGHUP reads the kernel key again. SIGTERM or SIGINT refuses new requests, lets running ones finish and stops; SIGTERM to a pair's primary hands its role to the standby in sync first. SIGUSR1 stops the same way, never handing over (a new binary takes the stores over). `--dev` is for trying kernable out: it creates `root.key`, `kernel.key`, `vault.key`, `app.key` and `app.grant` in the working directory if missing (app may do everything) and defaults to `--root root.pub`, `--vault-key vault.key` and `--ui-listen 127.0.0.1:7234`. Exit 2 on invalid flags; exit 1 if the kernel stops on an error. ### `kernable demo` ```text kernable demo [--ui-listen ] ``` Runs kernable for presentations: a kernel on a new store in a temporary directory, with a small bank built in, shown step by step on the Stage at `http:///demo` (default `127.0.0.1:7234`). Reset starts again on a new store; Ctrl-C ends it and removes everything. ## Keys and trust ### `kernable keys new` ```text kernable keys new ``` Writes `.key` (the secret key, readable by the owner only, never overwritten) and `.pub` (the key ID), and prints the key ID. ### `kernable grant` ```text kernable grant [--succeeds ] ... ``` Writes `.grant`, signed by the root key, and prints its permissions. A permission is `=,`... or a bare `` for every action. Actions: `submit` `run` `subscribe` `listen` `query` `answer` `check` `privileged` `vault_read` `vault_erase` `data_read` `data_write` `cache` `manage` `replicate` `export` `backup` `reserved_order`. A type is `*`, `.*` or `.`; manage takes service names, privileged takes `bypass_authorization`; replicate, export, backup and `reserved_order` (negative listener orders, for kits) take `*`. `--succeeds` lets this key take over the aggregates and listeners registered with ``. A newer grant for the same key replaces the older one. ### `kernable revoke` ```text kernable revoke ``` Revokes a key for good from `` on (UTC, like `2026-10-27T09:00:00Z`): the kernel refuses it, ends its connections and rejects its queued commands accepted since then (`key_revoked`). The kernel must be running. ### `kernable witness` ```text kernable witness --listen --root --state [--lease-ms ] [--tls-...] ``` Runs the witness of a pair of kernels started with `--witness`: it decides which one is the primary, renews the primary's lease (3000 ms by default) and lets the standby in sync take its place when the lease runs out. It keeps the term and the roles in `` (default witness.state); listens on `127.0.0.1:7300` by default, elsewhere only with mutual TLS (`--tls-cert`, `--tls-key`, `--tls-client-ca`). ### `kernable partitions apply` ```text kernable partitions apply ``` Applies a partition map on the control partition at ``: {'epoch': 2, 'partitions': {'p1': ['host:7233'], 'p2': [...]}, 'default': 'p1', 'control': 'p1', 'tenants': {'acme': 'p2'}} (JSON, with double quotes). Tenants not listed live on the default partition; the epoch must grow with every map. `` needs a grant for manage on `*`. The other partitions take it from the control partition. ### `kernable partitions show` ```text kernable partitions show ``` Prints the partition map the kernel at `` holds. ### `kernable partitions move` ```text kernable partitions move ``` Moves `` with its data to partition ``: applies a map that marks it moving on the control partition at ``, waits until `` has copied it, then applies the map with the tenant on ``. Meanwhile the tenant's requests are answered UNAVAILABLE. Run it again to finish a move it did not. `` needs manage on `*`. ### `kernable promote` ```text kernable promote ``` Makes the standby at `` (started with `--follow`) the primary: it stops following and serves. `` needs a grant for manage on `*`. Promote only a standby whose primary is gone: a primary that still runs is fenced (it tells nobody of a write from then on), and rejoins as a standby of the new one. Without a witness, a standby the primary never told it is in sync is refused: the primary may have told writes it lacks. ### `kernable switchover` ```text kernable switchover ``` Hands the primary role over to the standby in sync now: the primary at `` refuses new requests (the SDKs retry them on the other kernel), lets running ones finish, and waits until the standby holds every record (at most 10 s). In a pair (`--witness`) it tells the witness; without one it tells the standby on its stream. The standby serves at once, and the old primary stops: in a pair, started again, it follows; without a witness, it starts only with `--follow`. For updates and moves: update or start the standby, switch over, update or retire the old primary. `` needs a grant for manage on `*`. ### `kernable certs new` ```text kernable certs new ... ``` Writes a development CA (`ca.tls.crt`), a kernel certificate (`kernel.tls.crt`, `kernel.tls.key`) and a certificate per `` (`.tls.crt`, `.tls.key`) to ``, for mutual TLS. The CA key is not kept: issue a new set to add clients. Production brings its own PKI. ## Deploy and inspect ### `kernable apply` ```text kernable apply ``` Applies a service's membership: a JSON manifest with service, workers, listeners, middlewares, processes and metadata. A process the service's membership listed before and no longer lists starts no new cases. Signed with ``, whose grant needs `manage=`; the grant file next to the key (`app.grant` for `app.key`) is sent along. The kernel must be running. Once per deploy. ### `kernable services` ```text kernable services ``` Lists each applied membership: the service, its metadata, who applied it and when, and the workers, listeners and middlewares it lists, marking those that never registered; then the service's dynamic listeners, registered but not listed. ### `kernable workers` ```text kernable workers ``` Lists the registered aggregate types with their command types, key, service and when a worker for them was last seen. ### `kernable listeners` ```text kernable listeners ``` Lists the registered listeners with order, position, event types, key, service, middleware and when they were last seen. A listener its service's membership does not list is marked: it counts for nobody. ### `kernable listeners parked` ```text kernable listeners parked ``` Lists the events listeners parked: the listener, position, event type, code, attempts, when, whether a retry is asked for, and the message (for a handler error only the error's type; the listener's log has the rest). ### `kernable schemas` ```text kernable schemas ``` Lists the highest stored schema version per event type. ### `kernable processes list` ```text kernable processes list ``` Lists the registered business processes: name, newest version, state (active or removed), as whom it acts (ProcessService.Act; else `process:`), and when and by which key the newest version was registered. ### `kernable processes show` ```text kernable processes show [] ``` Prints a process's model as registered, the newest version or ``, as indented JSON. Exit 1 if there is no such process or version. ### `kernable processes cases` ```text kernable processes cases [--state running|waiting|incident|ended] ``` Lists the cases of a process: tenant, case ID, state (running: a command sent or a query asked; waiting: for an event or a time; incident; ended) and the step it is at (or how it ended). Reads every event, also with the kernel stopped; the UI's Processes view shows them live. ### `kernable processes case` ```text kernable processes case ``` Prints a case's events as a timeline: when, version, event type and its data. ## Processes, with the kernel running ### `kernable processes retry|skip|cancel` ```text kernable processes retry|skip|cancel [--tenant ] ``` Resolves a case's incident: retry runs its step again, skip goes on to the step's next step, cancel ends the case (also without an incident). Signed with ``, whose grant needs `process=`; the grant file next to the key is sent along. `--tenant` names the case's tenant when its ID is used in several. ### `kernable processes remove` ```text kernable processes remove ``` Removes a process: it starts no new cases, its running cases finish; registering it again makes it active. Signed with ``, whose grant needs `process=`; the grant file next to the key is sent along. Exit 1 if it was removed already. ## Parked events, with the kernel running ### `kernable listeners retry` ```text kernable listeners retry ``` Delivers the listener's parked event at `` again, or all of them, the next time it is connected; if it fails again, it is parked again. Signed with ``, whose grant needs `manage=` of the listener (`manage=*` for one without a service); the grant file next to the key is sent along. Exit 1 if none of them is parked. ### `kernable listeners dismiss` ```text kernable listeners dismiss ``` Removes the parked event at ``, or all of them, without delivering them again. Like retry otherwise. ## Check the store ### `kernable verify` ```text kernable verify ... ``` Recomputes the hash chain and checks every checkpoint and signature, with every key the kernel ever had: give all its `.pub` files. Read-only, so the kernel may run. Prints the counts, `PROBLEM` and `SUSPECT` lines, then `OK` or `TAMPERED`. Exit 0: intact. Exit 1: tampered, or the check failed. Exit 2: intact, but something was signed by a key at or after its revocation (`SUSPECT` lines). Events after the last checkpoint are counted: cutting them off could not be detected. ### `kernable checkpoint` ```text kernable checkpoint ``` Signs a checkpoint of the newest event now, as the running kernel does every `--checkpoint-interval-ms`. Kernel stopped. ## Export and import ### `kernable export` ```text kernable export ``` Writes everything the running kernel at `` holds to ``: every record of its store, every file of its data store and the vault values, which stay encrypted with its vault key. The kernel signs a checkpoint first. `` needs a grant for export on `*`; the grant file next to the key is sent along. `` must not exist; it is written readable by its owner only, and deleted if the export does not finish. Snapshots and the cache are not exported. Prints the counts and the kernel keys that signed the store. ### `kernable export` ```text kernable export [--store ] [--data-store ] [--vault-store --vault-key ] ``` The same from stores read directly, kernel stopped. The flags name the stores as for kernable start, with its defaults; without `--vault-key` the vault is left out, which fails if the store refers to vault values. ### `kernable import` ```text kernable import [--store ] [--data-store ] [--vault-store --vault-key ] [--kernel-pub ]... ``` Fills new, empty stores from an export, then checks the store like verify, with the same output and exit codes. `--vault-key` must be the source's vault key. The whole file is checked before anything is written: a copy cut short or changed, or a wrong vault key, is refused. Without `--kernel-pub`, the store is checked with the kernel keys the export names; compare them with the source's `.pub` files, or name those. Keep the source's kernel `.pub` files for verify after the kernel runs on the new stores. ## Run a node ### `kernable node` ```text kernable node ``` Runs one node of a project: the kernel and, with a [backup] section, the backup follower, as child processes it restarts when they exit. It reads the file again every two seconds or on SIGHUP, and restarts a child whose command changed or whose binary was replaced. SIGTERM or SIGINT stops the children (SIGKILL after stop-seconds, default 30). The file has [kernel] (flags: the flags of kernable start without their dashes, env, binary, stop-seconds), [backup] (to, key, backup-key, follow, segment-seconds, base-hours, keep, check-hours, env), [node] (instance, project, podman, metrics-listen: GET /metrics) and any number of [[service]] (name, build, file, command, key, env, secrets, cpus, memory, stop-seconds, writable, expose): containers built with podman on the node, rebuilt on SIGHUP, run after the kernel with `KERNABLE_URL`, `KERNABLE_INSTANCE` and their key. Every child's output comes out of the node prefixed with its name. Paths are relative to its directory. Exit 2 if the file is invalid at start. ### `kernable expose` ```text kernable expose =... Passes every connection to each Unix socket on to its ``` address, both ways: a node runs it in an instance's pod, for a service's port to be reached through the socket. ## Continuous backup ### `kernable backup key` ```text kernable backup key ``` Writes a new backup key (32 random bytes in hex) to ``, readable by its owner only; never overwrites one. Keep it apart from the backups, as you keep the root key: without it a backup cannot be read. ### `kernable backup` ```text kernable backup --follow [,] --key --to --backup-key [--segment-seconds ] [--base-hours ] [--keep ] [--check-hours ] [--status-file ] ``` Follows the kernel at `` (started with `--backup` on; both kernels of a pair, comma-separated) and writes its records to ``, encrypted with the backup key, until stopped: a base, then a segment every `--segment-seconds` (default 5) or 16 MiB, a new generation every `--base-hours` (default 24), the newest `--keep` (default 7) kept. `` needs a grant for backup on `*`; the grant file next to it is sent along. `` is `file:` or `s3:///`, with `KERNABLE_S3_ENDPOINT`, `KERNABLE_S3_REGION`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. Vault values stay encrypted with the kernel's vault key besides. With `--check-hours`, it also checks the backup that often, as backup check does, and logs what it found. With `--status-file`, it notes there (JSON) when it last uploaded, up to which record, and when it last checked and what it found. ### `kernable backup list` ```text kernable backup list --backup-key ``` Prints the generations of a backup: each base's record and time, and how far its segments reach. ### `kernable backup check` ```text kernable backup check --backup-key ``` Reads the newest generation all through, without writing a store: every object opens with the key and decodes, and the records chain from the base to the last segment. Prints the record a restore reaches and how old it is; exits 1 if the backup does not check out. ### `kernable restore` ```text kernable restore --backup-key [--at