Appearance
Monitor kernable
Use this page to see what a running kernel is doing: in a browser while you develop or debug, and with health checks, metrics and alerts in production. To find the cause of a specific problem, start at Failures and recovery.
| Want to | Use |
|---|---|
| see at a glance whether anything needs attention | the UI's Overview (--ui-listen) |
| see where the cases of a business process stand, and resolve its incidents | the UI's Processes, kernable processes |
| see services, lag, parked events, rejections, the standby, the store | the UI's Operations |
| see the domains, the log, one aggregate's history, or what the vault, snapshots, files and cache hold | the UI's Explore |
| gate traffic and deploys | GET /ready (--metrics-listen) |
| alert and graph | GET /metrics (--metrics-listen) |
| list registrations with the kernel stopped or running | kernable services, kernable workers, kernable listeners |
The UI
With --ui-listen (on by default with --dev, at http://127.0.0.1:7234), the kernel shows itself in a browser. The start banner shows its address in the UI line, or disabled. The top bar has the four places below, a search box, and the verdict: Healthy, Degraded or the number of problems, with the reasons on hover. The page does not ask over and over: the kernel tells it when the log moved on or the verdict changed (server-sent events), and the page then reloads what it shows, at most every 2 s; without that connection it asks every 5 s. Every view has its own address (/processes/shop.fulfilment, /processes/shop.fulfilment/cases/o-1 for the newest case of order o-1, /explore/events?type=shop.Order; another tenant than default as ?tenant=), so a reload keeps it and a link can be sent to someone with the same tunnel.
Overview: commands and events over the kernel's run (up to a day, counted once a minute in memory: a restart starts over), cases started and open, incidents and rejected commands; a chart of activity per 10 minutes; Needs attention, every problem in one list with what fixes it (a listener missing or behind, parked events, incidents, a standby not in sync, a backup follower away, the store near its quota, a burst of rejections); the processes with their outcomes; and the system at a glance.
Processes: each business process, one version at a time, over the last hour, day or week:
- figures: cases started, ended per end with their share, the median and 95th percentile at the first wait, open cases and incidents;
- the diagram, each path as wide as the number of cases that took it, with its count and share where a step branches, and the cases at each step now (incidents in red). It runs from left to right while it fits and from top to bottom when the process is too long; Direction sets it for good (kept in your browser). The conditions of the decisions are listed under it, and the ends have neutral colors, as no end is good or bad by itself. Click a step to list its cases;
- outcomes over time, and the open cases by how long they have waited at their step, with the oldest one;
- Cases, newest first, filtered by state, step and case, key or tenant; Incidents with the commands that resolve them (
kernable processes retry|skip|cancel, ready to copy); and the statistics per step.
A case has its own address (
/processes/<process>/cases/<key>, with/<n>for an earlier case of the same key). It opens with an incident first, if it has one, with the commands that resolve it; then the diagram showing its own way (the steps it reached and how long it stayed at each, the rest faded) and as a story in plain words: what started it, what it waited for, what it sent and with what outcome, where it ended, each linked to the aggregates it touched; its raw events are one click away. The UI counts from the case events in the log, starting when you first open it: on a large log it says how far it has read until it has caught up. It keeps the running cases and the newest 1,000 ended ones per process, and its statistics for a week.Operations: the kernel (version, uptime, queued and scheduled commands), the standby and witness, the backup follower, the store's size against
--max-store-bytes, what limits refused; every worker and listener with its key, how far it was behind over the last hour and its parked events; Parked events with the command that delivers each again; and Rejected commands grouped by code, with why (the unique key taken, the requirement that did not hold). The process engine's own refused commands (a timer no longer needed) are left out unless you ask for them. Commands the kernel refuses before it accepts them (a bad signature, a missing grant) are not stored and do not show here; the kernel's log has them.Explore: for developers.
- Domains: totals over every domain (events in the log and in the last hour, commands in the last hour and the share rejected, open incidents), then one card per domain, by name or busiest first: what it holds (aggregates, commands, events, queries, processes), its numbers, its workers, and what needs attention (an aggregate without a worker, a reader away or behind, incidents in its processes), with links to its flow, its events and commands in the logs, and its part of the catalog. A domain opens with its numbers (events in the log, events and commands in the last hour with the share rejected, the last event), then per aggregate its commands (how many came in the last hour, how many were rejected, which processes send them) beside the events it writes (how many the log holds, the last hour per minute, the last one, who reads them); then the read models, listeners and middleware concerned with how far each is behind, its processes, and the events it uses from other domains. The UI counts the events the first time it opens a domain, from the start of the log, and from then on as they come.
- Catalog: every event type, command, query and process, grouped by domain, in one table to filter by text or domain; a click on a column header sorts by it (then as one list, the domain before each name), a click on the first column back by domain and name. Events: who writes and who reads them, how many the log holds, the last hour per minute and the last one. Commands: how many came in the last hour and how many were rejected, who checks them and which processes send them; a domain whose aggregate has no worker says so. Queries: how many were asked in the last hour, how many were not answered (rejected, or no read model answered in time) and how long an answer took on average, the read model that answers, what it reads and how far it is behind; these counts start when the kernel starts. Processes: the event they start on, what they wait for, send and ask, their open cases and incidents, and how many started. A row opens the event log, the command log or the process.
- Event log: the whole log, newest first under a heading per day, each event with who sent the command behind it (from the request context the event carries, no lookup needed), filtered by domain, event type, aggregate ID, command or correlation ID, text in the payload and tenant, without the process engine's events unless you include them. The filters in use show as chips to remove; the address keeps them. Events one command wrote together are marked on the left; the tenant column shows once more than one tenant is among them. Live shows new ones as they commit, highlighted for a moment. An event opens with its payload beside where it came from (sender, correlation, cause, case, command, worker key, hash) and links to the aggregate's history, the events of the same correlation, the command and the case. One search reads at most 20,000 events, so on a large log it may stop early and say how far back it got.
- Command log: every stored command, newest first: when it was received, the aggregate, who sent it (a person, a service, or a process and its case), and what came of it: committed with the events it wrote and how long it took from receipt (or its due time) to the commit, rejected with the code and why, waiting, or scheduled for when. Commands come under a heading per day; those of one correlation next to each other are marked on the left. Filter by domain, type, aggregate ID, sender, command or correlation ID, outcome and tenant (the filters in use show as chips to remove, and the address keeps them); the process engine's own commands only on request; Live shows new ones, highlighted for a moment, and updates waiting ones. Open one for its payload beside who sent it and what came of it, with links to the aggregate's history, its events, the commands of the same correlation and the case.
- Data: the stores beside the log, counted and listed but never a value: the UI does not decrypt the vault, show snapshot states or open files.
- Vault: personal and secret values, how many subjects hold them and per aggregate type, the erasures (the vault keeps only a hash of the subject) and those not carried out yet, and how many values the vault key has encrypted against the 2³² one key may. Past half of that the verdict warns: plan a
kernable vault rekey(The vault's key). A subject is looked up by its exact ID: its values per aggregate type and its erasures. Subjects are not listed. - Snapshots: per aggregate type how many, their size and schemas, when the last was taken, and how many events the aggregates have gained since, with the aggregates furthest from their snapshot (
snapshotEvery). - Files: per tenant and domain how many files and their size; a domain opens as folders by
/in the keys, with each file's content type and size (Store files and cache values). - Cache: entries and memory against
--max-cache-bytes, the hit rate, the values that expired or were evicted to make room, per domain, and the keys with how long until they expire. All of it starts with the kernel, as the cache does.
- Vault: personal and secret values, how many subjects hold them and per aggregate type, the erasures (the vault keeps only a hash of the subject) and those not carried out yet, and how many values the vault key has encrypted against the 2³² one key may. Past half of that the verdict warns: plan a
- Aggregates: without one open, the aggregate types with how many there are and when one last changed; the list filters by type and by ID or tenant. An aggregate opens with its version, when it was created and last changed, its unique keys, facts and snapshot, the process cases it started, and links to its events and commands in the logs. Its events follow as a timeline under a heading per day, with the time between them, each with the command and sender that caused it and a color per correlation, so the events of one chain of commands share it; new ones appear live. One click opens the payload and the command beside where it came from and its signatures. Vault fields show as
personal · <subject>,secretorerased; the UI never decrypts the vault, and it does not show snapshot states, which may hold personal values.
Search: an order, case, command or correlation ID finds its aggregates, its cases and its events.
Keys show by their ID. Click one for its full ID (to copy), what it is used for now (the workers, listeners and processes it signs), and a name such as "app" or "operator" to show instead, kept in your browser only.
The UI has no login, so it listens on a loopback address only; the kernel refuses to start with another address. Reach it on a server through an SSH tunnel:
sh
ssh -L 7234:127.0.0.1:7234 host # then open http://127.0.0.1:7234If the address is taken (another kernel in development), the kernel runs on without the UI and logs the warning the UI is off.
Watching from a script
The UI's own stream is open to anything on the machine, like the UI: server-sent events at /api/watch, read-only and without login, on the loopback address only. Choose what to get with topics and narrow it with filters:
sh
curl -N 'http://127.0.0.1:7234/api/watch?topics=events&domain=shop&type=order&id=o-1' # one order's events
curl -N 'http://127.0.0.1:7234/api/watch?topics=commands&status=rejected' # rejected commands
curl -N 'http://127.0.0.1:7234/api/watch?topics=cases&process=shop.fulfilment' # a process's cases
curl -N 'http://127.0.0.1:7234/api/watch?topics=health,pulse' # the verdict, counts, listener positions| Topic | Each message |
|---|---|
events | the new events that match, with their sender and data (vault fields as references only) |
commands | commands that match when stored, and again when they complete: committed with their events and duration, rejected with why |
cases | the case events of a process (process=, case=) |
head | the log moved on, with the aggregates the new events changed |
health | the verdict, when it changed |
pulse | the head, the counts and each listener's position, when any changed |
Filters: tenant, domain, type (aggregate type), id (aggregate ID), event (part of domain.EventType), command (command or correlation ID), sender (part of the identity), process, case, status (of commands); the process engine's own events and commands only with engine=1. Messages come at most once a second per topic. events and cases carry the log position as their id: connected again with Last-Event-ID (browsers do it by themselves) or from=<position>, the stream goes on from there. More than 500 events behind, it sends gap instead: read what you missed from /api/events. For services elsewhere, subscribe through the SDKs instead: they are signed and checked against grants.
Health checks
Set --metrics-listen (for example 127.0.0.1:9233). It serves plain HTTP with no login: bind it to a private address. On a loopback address it answers only requests whose Host is a loopback name (localhost, 127.0.0.1, [::1]), as the UI does, so a web page cannot read it through DNS rebinding; health checks and Prometheus on the same machine send one anyway.
Liveness:
GET /healthanswers 200okwhile the process runs.Readiness:
GET /readyanswers 200readywhen the kernel can serve, and 503 with the reason otherwise:503 body Meaning waiting for workers: shop.Cartno worker is connected for these aggregates waiting for listeners: carts, guardthese listeners or middlewares are not connected, or have not caught up yet shutting downthe kernel got SIGTERM and finishes running requests It waits for what counts: what a service's membership lists, also members that never registered, and for services without a membership everything they registered. Dynamic members (registered, but not listed) and live listeners do not count (Order and consistency). Use it to gate traffic and deploys (Deploy and upgrade).
Alerts worth having
yaml
groups:
- name: kernable
rules:
- alert: KernableNotReady # a worker or listener that counts is missing or behind
expr: kernable_ready == 0
for: 2m
- alert: KernableListenerLagging # a listener falls behind the log
expr: kernable_listener_lag > 1000
for: 5m
- alert: KernableListenerDown
expr: kernable_listener_connected == 0
for: 2m
- alert: KernableEventsParked # a reactor gave up on events: retry or dismiss them
expr: kernable_listener_parked > 0
for: 1m
- alert: KernableNoWorkers # commands queue up with nobody to run them
expr: kernable_worker_sessions == 0 and kernable_queued_commands > 0
for: 1m
- alert: KernableMiddlewareFailing # checks refused because a middleware is away or slow
expr: rate(kernable_middleware_total{outcome="failed"}[5m]) > 0
for: 5m
- alert: KernableQueriesFailing # no read model answered in time
expr: rate(kernable_queries_total{outcome="failed"}[5m]) > 0
for: 5m
- alert: KernableLimitReached # requests refused by a limit (--max-*)
expr: rate(kernable_refused_total[5m]) > 0
for: 5m
- alert: KernableStoreFilling # the stores passed 80 % of --max-store-bytes
expr: kernable_store_bytes / kernable_store_bytes_max > 0.8
for: 15mkernable_refused_totalnames the limit in itsreasonlabel (Limits). A growingqueued_commandsorqueued_per_aggregateusually means a worker is away;requests_per_keya client that does not wait for its answers;handshake_timeoutorconnectionssomething that opens connections and never uses them.rateis a key that starts more requests per second than--max-requests-per-second-per-key;quota_storagea kernel whose stores reached--max-store-bytes, which refuses new commands and file puts until data is deleted or the limit raised.- The per-listener metrics (
kernable_listener_lag,kernable_listener_connected) cover every listener the kernel knows, also dynamic ones that nobody waits for. Exclude those by theirlistenerlabel, or remove them (Remove a listener). - A listener stuck at an event it cannot handle (a throwing handler, or a newer event version) shows as lag that keeps growing while the listener stays connected (When a listener is slow or stuck).
Useful views
| Query | Shows |
|---|---|
sum by (outcome) (rate(kernable_commands_total[5m])) | throughput and the share of rejections |
rate(kernable_execution_seconds_sum[5m]) / rate(kernable_execution_seconds_count[5m]) | how long a command takes from lease to commit |
rate(kernable_settle_seconds_sum[5m]) / rate(kernable_settle_seconds_count[5m]) | how long submits wait for listeners |
rate(kernable_query_seconds_sum[5m]) / rate(kernable_query_seconds_count[5m]) | query latency through the kernel |
rate(kernable_middleware_seconds_sum[5m]) / rate(kernable_middleware_seconds_count[5m]) | time spent in middleware checks |
kernable_queued_commands | commands waiting or running |
The timings are summaries without quantiles: they give averages. All metrics are in the reference.
From the command line
These read the store, whether the kernel runs or not; the UI shows the live state:
sh
kernable services sqlite:shop.db # each membership: who applied it, what it lists, (never registered), dynamic listeners
kernable workers sqlite:shop.db # aggregate, command types, key, service, last seen
kernable listeners sqlite:shop.db # name, order, position, types, key, service, middleware, parked, last seen
kernable listeners parked sqlite:shop.db # each parked event: listener, position, type, code, attempts, when, messagelast seen tells you when a worker or listener was last connected. See the CLI reference.
Usage beside the store
The kernel writes <store>.usage beside its store (kernable.db.usage) every minute and when it stops, one line of JSON:
json
{"at":1793996832409,"events":205200,"commands":205211,"rejected":3,"queries":1200,"store_bytes":241172480,"sig":"9f3c…"}events is the store's head, store_bytes what its stores take on the disk; commands (committed), rejected and queries (answered) go on across restarts: a kernel starts from the counts in the file, so a crash loses at most a minute of them. For whoever bills or watches the machine without scraping metrics. sig is the kernel key's signature (Ed25519, hex) over the other fields, one per line after the tag kernable-usage-1, in the order at, events, commands, rejected, queries, store_bytes, replication (empty without replication): checked with the kernel's public key, the counts are the kernel's, not another process's on the machine.
Parked events
A reactor that gives up on an event parks it and moves on (Failures): its follow-up was rejected, or its handler failed five times. The kernel keeps the event's position with the reason until you act on it; the UI shows the count on the listener, the first ones in its details and all of them under Connections, and kernable_listener_parked counts them.
sh
kernable listeners parked sqlite:shop.db
# confirmations position 18204 shop.OrderSubmitted not_found attempts 1 parked 2026-11-02T14:03:00Z: no such customer
# mailer position 18517 shop.OrderSubmitted handler_error attempts 5 parked 2026-11-02T14:21:07Z: TypeErrorFor a handler error the message is only the error's type; the reactor's log has the whole error, at the time it parked the event. Fix the cause, deploy, then deliver the events again, or drop those that no longer matter:
sh
kernable listeners retry 127.0.0.1:7233 ops.key mailer 18517 # or --all
kernable listeners dismiss 127.0.0.1:7233 ops.key confirmations --allThe key needs manage for the listener's service. A retried event goes to the reactor on its own, before anything new, the next time it is connected; if it fails again, it is parked again. A follow-up command keeps its fixed ID, so retrying a rejected one returns the same rejection: retry once the handler deals with it differently, or dismiss it.
Logs
The kernel logs to stderr. KERNABLE_LOG sets the filter (default info; e.g. debug or warn) and KERNABLE_LOG_FORMAT=json switches to JSON for your log pipeline. The start banner appears only on a terminal, and not with JSON logs. Watch for key revoked and drain timed out warnings.
The SDKs print what their workers and listeners do to stderr, one line each:
text
12:04:10 worker connected to http://127.0.0.1:7233 · key app (69212a19f36f…) · runs shop.AddItemThere, KERNABLE_LOG is off, error, warn, info (default) or debug. Lines have colors on a terminal unless NO_COLOR is set. An app prints an overview of its workers and listeners once all are connected.
