Previewkernable is not released yet: these docs describe work in progress, and things may change before 1.0.Not released yet.Request early access →
Skip to content

Monitor kernable ​

Use this page to see what a running kernel is doing: in a browser while you develop or debug, and with health checks, metrics and alerts in production. To find the cause of a specific problem, start at Failures and recovery.

Want toUse
see at a glance whether anything needs attentionthe UI's Overview (--ui-listen)
see where the cases of a business process stand, and resolve its incidentsthe UI's Processes, kernable processes
see services, lag, parked events, rejections, the standby, the storethe UI's Operations
see the domains, the log, one aggregate's history, or what the vault, snapshots, files and cache holdthe UI's Explore
gate traffic and deploysGET /ready (--metrics-listen)
alert and graphGET /metrics (--metrics-listen)
list registrations with the kernel stopped or runningkernable services, kernable workers, kernable listeners

The UI ​

With --ui-listen (on by default with --dev, at http://127.0.0.1:7234), the kernel shows itself in a browser. The start banner shows its address in the UI line, or disabled. The top bar has the four places below, a search box, and the verdict: Healthy, Degraded or the number of problems, with the reasons on hover. The page does not ask over and over: the kernel tells it when the log moved on or the verdict changed (server-sent events), and the page then reloads what it shows, at most every 2 s; without that connection it asks every 5 s. Every view has its own address (/processes/shop.fulfilment, /processes/shop.fulfilment/cases/o-1 for the newest case of order o-1, /explore/events?type=shop.Order; another tenant than default as ?tenant=), so a reload keeps it and a link can be sent to someone with the same tunnel.

  • Overview: commands and events over the kernel's run (up to a day, counted once a minute in memory: a restart starts over), cases started and open, incidents and rejected commands; a chart of activity per 10 minutes; Needs attention, every problem in one list with what fixes it (a listener missing or behind, parked events, incidents, a standby not in sync, a backup follower away, the store near its quota, a burst of rejections); the processes with their outcomes; and the system at a glance.

  • Processes: each business process, one version at a time, over the last hour, day or week:

    • figures: cases started, ended per end with their share, the median and 95th percentile at the first wait, open cases and incidents;
    • the diagram, each path as wide as the number of cases that took it, with its count and share where a step branches, and the cases at each step now (incidents in red). It runs from left to right while it fits and from top to bottom when the process is too long; Direction sets it for good (kept in your browser). The conditions of the decisions are listed under it, and the ends have neutral colors, as no end is good or bad by itself. Click a step to list its cases;
    • outcomes over time, and the open cases by how long they have waited at their step, with the oldest one;
    • Cases, newest first, filtered by state, step and case, key or tenant; Incidents with the commands that resolve them (kernable processes retry|skip|cancel, ready to copy); and the statistics per step.

    A case has its own address (/processes/<process>/cases/<key>, with /<n> for an earlier case of the same key). It opens with an incident first, if it has one, with the commands that resolve it; then the diagram showing its own way (the steps it reached and how long it stayed at each, the rest faded) and as a story in plain words: what started it, what it waited for, what it sent and with what outcome, where it ended, each linked to the aggregates it touched; its raw events are one click away. The UI counts from the case events in the log, starting when you first open it: on a large log it says how far it has read until it has caught up. It keeps the running cases and the newest 1,000 ended ones per process, and its statistics for a week.

  • Operations: the kernel (version, uptime, queued and scheduled commands), the standby and witness, the backup follower, the store's size against --max-store-bytes, what limits refused; every worker and listener with its key, how far it was behind over the last hour and its parked events; Parked events with the command that delivers each again; and Rejected commands grouped by code, with why (the unique key taken, the requirement that did not hold). The process engine's own refused commands (a timer no longer needed) are left out unless you ask for them. Commands the kernel refuses before it accepts them (a bad signature, a missing grant) are not stored and do not show here; the kernel's log has them.

  • Explore: for developers.

    • Domains: totals over every domain (events in the log and in the last hour, commands in the last hour and the share rejected, open incidents), then one card per domain, by name or busiest first: what it holds (aggregates, commands, events, queries, processes), its numbers, its workers, and what needs attention (an aggregate without a worker, a reader away or behind, incidents in its processes), with links to its flow, its events and commands in the logs, and its part of the catalog. A domain opens with its numbers (events in the log, events and commands in the last hour with the share rejected, the last event), then per aggregate its commands (how many came in the last hour, how many were rejected, which processes send them) beside the events it writes (how many the log holds, the last hour per minute, the last one, who reads them); then the read models, listeners and middleware concerned with how far each is behind, its processes, and the events it uses from other domains. The UI counts the events the first time it opens a domain, from the start of the log, and from then on as they come.
    • Catalog: every event type, command, query and process, grouped by domain, in one table to filter by text or domain; a click on a column header sorts by it (then as one list, the domain before each name), a click on the first column back by domain and name. Events: who writes and who reads them, how many the log holds, the last hour per minute and the last one. Commands: how many came in the last hour and how many were rejected, who checks them and which processes send them; a domain whose aggregate has no worker says so. Queries: how many were asked in the last hour, how many were not answered (rejected, or no read model answered in time) and how long an answer took on average, the read model that answers, what it reads and how far it is behind; these counts start when the kernel starts. Processes: the event they start on, what they wait for, send and ask, their open cases and incidents, and how many started. A row opens the event log, the command log or the process.
    • Event log: the whole log, newest first under a heading per day, each event with who sent the command behind it (from the request context the event carries, no lookup needed), filtered by domain, event type, aggregate ID, command or correlation ID, text in the payload and tenant, without the process engine's events unless you include them. The filters in use show as chips to remove; the address keeps them. Events one command wrote together are marked on the left; the tenant column shows once more than one tenant is among them. Live shows new ones as they commit, highlighted for a moment. An event opens with its payload beside where it came from (sender, correlation, cause, case, command, worker key, hash) and links to the aggregate's history, the events of the same correlation, the command and the case. One search reads at most 20,000 events, so on a large log it may stop early and say how far back it got.
    • Command log: every stored command, newest first: when it was received, the aggregate, who sent it (a person, a service, or a process and its case), and what came of it: committed with the events it wrote and how long it took from receipt (or its due time) to the commit, rejected with the code and why, waiting, or scheduled for when. Commands come under a heading per day; those of one correlation next to each other are marked on the left. Filter by domain, type, aggregate ID, sender, command or correlation ID, outcome and tenant (the filters in use show as chips to remove, and the address keeps them); the process engine's own commands only on request; Live shows new ones, highlighted for a moment, and updates waiting ones. Open one for its payload beside who sent it and what came of it, with links to the aggregate's history, its events, the commands of the same correlation and the case.
    • Data: the stores beside the log, counted and listed but never a value: the UI does not decrypt the vault, show snapshot states or open files.
      • Vault: personal and secret values, how many subjects hold them and per aggregate type, the erasures (the vault keeps only a hash of the subject) and those not carried out yet, and how many values the vault key has encrypted against the 2³² one key may. Past half of that the verdict warns: plan a kernable vault rekey (The vault's key). A subject is looked up by its exact ID: its values per aggregate type and its erasures. Subjects are not listed.
      • Snapshots: per aggregate type how many, their size and schemas, when the last was taken, and how many events the aggregates have gained since, with the aggregates furthest from their snapshot (snapshotEvery).
      • Files: per tenant and domain how many files and their size; a domain opens as folders by / in the keys, with each file's content type and size (Store files and cache values).
      • Cache: entries and memory against --max-cache-bytes, the hit rate, the values that expired or were evicted to make room, per domain, and the keys with how long until they expire. All of it starts with the kernel, as the cache does.
    • Aggregates: without one open, the aggregate types with how many there are and when one last changed; the list filters by type and by ID or tenant. An aggregate opens with its version, when it was created and last changed, its unique keys, facts and snapshot, the process cases it started, and links to its events and commands in the logs. Its events follow as a timeline under a heading per day, with the time between them, each with the command and sender that caused it and a color per correlation, so the events of one chain of commands share it; new ones appear live. One click opens the payload and the command beside where it came from and its signatures. Vault fields show as personal · <subject>, secret or erased; the UI never decrypts the vault, and it does not show snapshot states, which may hold personal values.
  • Search: an order, case, command or correlation ID finds its aggregates, its cases and its events.

Keys show by their ID. Click one for its full ID (to copy), what it is used for now (the workers, listeners and processes it signs), and a name such as "app" or "operator" to show instead, kept in your browser only.

The UI has no login, so it listens on a loopback address only; the kernel refuses to start with another address. Reach it on a server through an SSH tunnel:

sh
ssh -L 7234:127.0.0.1:7234 host   # then open http://127.0.0.1:7234

If the address is taken (another kernel in development), the kernel runs on without the UI and logs the warning the UI is off.

Watching from a script ​

The UI's own stream is open to anything on the machine, like the UI: server-sent events at /api/watch, read-only and without login, on the loopback address only. Choose what to get with topics and narrow it with filters:

sh
curl -N 'http://127.0.0.1:7234/api/watch?topics=events&domain=shop&type=order&id=o-1'   # one order's events
curl -N 'http://127.0.0.1:7234/api/watch?topics=commands&status=rejected'               # rejected commands
curl -N 'http://127.0.0.1:7234/api/watch?topics=cases&process=shop.fulfilment'          # a process's cases
curl -N 'http://127.0.0.1:7234/api/watch?topics=health,pulse'                           # the verdict, counts, listener positions
TopicEach message
eventsthe new events that match, with their sender and data (vault fields as references only)
commandscommands that match when stored, and again when they complete: committed with their events and duration, rejected with why
casesthe case events of a process (process=, case=)
headthe log moved on, with the aggregates the new events changed
healththe verdict, when it changed
pulsethe head, the counts and each listener's position, when any changed

Filters: tenant, domain, type (aggregate type), id (aggregate ID), event (part of domain.EventType), command (command or correlation ID), sender (part of the identity), process, case, status (of commands); the process engine's own events and commands only with engine=1. Messages come at most once a second per topic. events and cases carry the log position as their id: connected again with Last-Event-ID (browsers do it by themselves) or from=<position>, the stream goes on from there. More than 500 events behind, it sends gap instead: read what you missed from /api/events. For services elsewhere, subscribe through the SDKs instead: they are signed and checked against grants.

Health checks ​

Set --metrics-listen (for example 127.0.0.1:9233). It serves plain HTTP with no login: bind it to a private address. On a loopback address it answers only requests whose Host is a loopback name (localhost, 127.0.0.1, [::1]), as the UI does, so a web page cannot read it through DNS rebinding; health checks and Prometheus on the same machine send one anyway.

  • Liveness: GET /health answers 200 ok while the process runs.

  • Readiness: GET /ready answers 200 ready when the kernel can serve, and 503 with the reason otherwise:

    503 bodyMeaning
    waiting for workers: shop.Cartno worker is connected for these aggregates
    waiting for listeners: carts, guardthese listeners or middlewares are not connected, or have not caught up yet
    shutting downthe kernel got SIGTERM and finishes running requests

    It waits for what counts: what a service's membership lists, also members that never registered, and for services without a membership everything they registered. Dynamic members (registered, but not listed) and live listeners do not count (Order and consistency). Use it to gate traffic and deploys (Deploy and upgrade).

Alerts worth having ​

yaml
groups:
  - name: kernable
    rules:
      - alert: KernableNotReady          # a worker or listener that counts is missing or behind
        expr: kernable_ready == 0
        for: 2m
      - alert: KernableListenerLagging   # a listener falls behind the log
        expr: kernable_listener_lag > 1000
        for: 5m
      - alert: KernableListenerDown
        expr: kernable_listener_connected == 0
        for: 2m
      - alert: KernableEventsParked      # a reactor gave up on events: retry or dismiss them
        expr: kernable_listener_parked > 0
        for: 1m
      - alert: KernableNoWorkers         # commands queue up with nobody to run them
        expr: kernable_worker_sessions == 0 and kernable_queued_commands > 0
        for: 1m
      - alert: KernableMiddlewareFailing # checks refused because a middleware is away or slow
        expr: rate(kernable_middleware_total{outcome="failed"}[5m]) > 0
        for: 5m
      - alert: KernableQueriesFailing    # no read model answered in time
        expr: rate(kernable_queries_total{outcome="failed"}[5m]) > 0
        for: 5m
      - alert: KernableLimitReached      # requests refused by a limit (--max-*)
        expr: rate(kernable_refused_total[5m]) > 0
        for: 5m
      - alert: KernableStoreFilling      # the stores passed 80 % of --max-store-bytes
        expr: kernable_store_bytes / kernable_store_bytes_max > 0.8
        for: 15m
  • kernable_refused_total names the limit in its reason label (Limits). A growing queued_commands or queued_per_aggregate usually means a worker is away; requests_per_key a client that does not wait for its answers; handshake_timeout or connections something that opens connections and never uses them. rate is a key that starts more requests per second than --max-requests-per-second-per-key; quota_storage a kernel whose stores reached --max-store-bytes, which refuses new commands and file puts until data is deleted or the limit raised.
  • The per-listener metrics (kernable_listener_lag, kernable_listener_connected) cover every listener the kernel knows, also dynamic ones that nobody waits for. Exclude those by their listener label, or remove them (Remove a listener).
  • A listener stuck at an event it cannot handle (a throwing handler, or a newer event version) shows as lag that keeps growing while the listener stays connected (When a listener is slow or stuck).

Useful views ​

QueryShows
sum by (outcome) (rate(kernable_commands_total[5m]))throughput and the share of rejections
rate(kernable_execution_seconds_sum[5m]) / rate(kernable_execution_seconds_count[5m])how long a command takes from lease to commit
rate(kernable_settle_seconds_sum[5m]) / rate(kernable_settle_seconds_count[5m])how long submits wait for listeners
rate(kernable_query_seconds_sum[5m]) / rate(kernable_query_seconds_count[5m])query latency through the kernel
rate(kernable_middleware_seconds_sum[5m]) / rate(kernable_middleware_seconds_count[5m])time spent in middleware checks
kernable_queued_commandscommands waiting or running

The timings are summaries without quantiles: they give averages. All metrics are in the reference.

From the command line ​

These read the store, whether the kernel runs or not; the UI shows the live state:

sh
kernable services sqlite:shop.db    # each membership: who applied it, what it lists, (never registered), dynamic listeners
kernable workers sqlite:shop.db     # aggregate, command types, key, service, last seen
kernable listeners sqlite:shop.db   # name, order, position, types, key, service, middleware, parked, last seen
kernable listeners parked sqlite:shop.db   # each parked event: listener, position, type, code, attempts, when, message

last seen tells you when a worker or listener was last connected. See the CLI reference.

Usage beside the store ​

The kernel writes <store>.usage beside its store (kernable.db.usage) every minute and when it stops, one line of JSON:

json
{"at":1793996832409,"events":205200,"commands":205211,"rejected":3,"queries":1200,"store_bytes":241172480,"sig":"9f3c…"}

events is the store's head, store_bytes what its stores take on the disk; commands (committed), rejected and queries (answered) go on across restarts: a kernel starts from the counts in the file, so a crash loses at most a minute of them. For whoever bills or watches the machine without scraping metrics. sig is the kernel key's signature (Ed25519, hex) over the other fields, one per line after the tag kernable-usage-1, in the order at, events, commands, rejected, queries, store_bytes, replication (empty without replication): checked with the kernel's public key, the counts are the kernel's, not another process's on the machine.

Parked events ​

A reactor that gives up on an event parks it and moves on (Failures): its follow-up was rejected, or its handler failed five times. The kernel keeps the event's position with the reason until you act on it; the UI shows the count on the listener, the first ones in its details and all of them under Connections, and kernable_listener_parked counts them.

sh
kernable listeners parked sqlite:shop.db
# confirmations position 18204 shop.OrderSubmitted not_found attempts 1 parked 2026-11-02T14:03:00Z: no such customer
# mailer position 18517 shop.OrderSubmitted handler_error attempts 5 parked 2026-11-02T14:21:07Z: TypeError

For a handler error the message is only the error's type; the reactor's log has the whole error, at the time it parked the event. Fix the cause, deploy, then deliver the events again, or drop those that no longer matter:

sh
kernable listeners retry 127.0.0.1:7233 ops.key mailer 18517     # or --all
kernable listeners dismiss 127.0.0.1:7233 ops.key confirmations --all

The key needs manage for the listener's service. A retried event goes to the reactor on its own, before anything new, the next time it is connected; if it fails again, it is parked again. A follow-up command keeps its fixed ID, so retrying a rejected one returns the same rejection: retry once the handler deals with it differently, or dismiss it.

Logs ​

The kernel logs to stderr. KERNABLE_LOG sets the filter (default info; e.g. debug or warn) and KERNABLE_LOG_FORMAT=json switches to JSON for your log pipeline. The start banner appears only on a terminal, and not with JSON logs. Watch for key revoked and drain timed out warnings.

The SDKs print what their workers and listeners do to stderr, one line each:

text
12:04:10  worker  connected to http://127.0.0.1:7233 · key app (69212a19f36f…) · runs shop.AddItem

There, KERNABLE_LOG is off, error, warn, info (default) or debug. Lines have colors on a terminal unless NO_COLOR is set. An app prints an overview of its workers and listeners once all are connected.