Previewkernable is not released yet: these docs describe work in progress, and things may change before 1.0.Not released yet.Request early access →
Skip to content

Metrics and health ​

With --metrics-listen 127.0.0.1:9233 the kernel serves plain HTTP with no login on that address, so bind it to a private one. How to use it for health checks, alerts and graphs: Monitor kernable.

  • GET /metrics: Prometheus text format, below.
  • GET /health: 200 while the process runs.
  • GET /ready: 200 once every aggregate that counts has a worker connected and every listener that counts is connected and has caught up (see Who counts), else 503 naming the workers or listeners it waits for.

Counters and durations count since the kernel started and begin at 0 again after a restart. Durations are summaries without quantiles, in seconds: <name>_sum and <name>_count. The outcome label takes the values named in the description; the listener label is a listener's name, for every registered listener, connected or not.

MetricTypeLabelsDescription
kernable_commands_totalcounteroutcomeCommands completed since the kernel started, by outcome: committed, or rejected by a worker or the kernel (scheduled commands cancelled, expired, or refused by a middleware when due included). Commands a middleware refused before the queue are not counted.
kernable_queries_totalcounteroutcomeQueries since the kernel started, by outcome: answered; rejected by a read model or a middleware; failed when no read model answered, because none was connected or none answered in time.
kernable_middleware_totalcounteroutcome, reasonRuns of a middleware chain, before or after a command or query, by outcome: passed, rejected, or failed when a middleware in the chain was not connected or did not answer within --middleware-timeout-ms.
kernable_refused_totalcounterRequests and connections a limit refused since the kernel started, by reason: connections, requests_per_key, subscriptions_per_key, listeners_per_key, queued_commands, queued_per_aggregate, queued_bytes, cache_bytes, handshake_timeout, too_large, http_host, http_connections.
kernable_queued_commandsgaugeCommands stored in the queue and not completed yet: waiting for a worker or running.
kernable_scheduled_commandsgaugeScheduled commands (deliver_at) not due yet, or due and being checked before they join their aggregate's queue; counted in kernable_queued_commands too.
kernable_worker_sessionsgaugeWorker sessions connected now.
kernable_head_positiongaugeGlobal position of the newest stored event; 0 while the store is empty.
kernable_execution_secondssummarySeconds from handing a command to a worker until its events are committed: the worker round trip plus storage. Committed commands only.
kernable_settle_secondssummarySeconds from the commit until every listener that counts acknowledged the events. Settling submits that settled in time only.
kernable_query_secondssummarySeconds from handing a query to the read models until one answered or rejected it. Queries that failed are not included.
kernable_middleware_secondssummarySeconds one middleware chain took to pass, reject or fail a request.
kernable_listener_positiongaugelistenerGlobal position of the last event the listener acknowledged.
kernable_listener_laggaugelistenerGlobal positions between the newest stored event and the listener's position: how far it is behind.
kernable_listener_connectedgaugelistener1 while the listener is connected, 0 while it is registered but away.
kernable_listener_parkedgaugeEvents the listener gave up on and parked, waiting for an operator to retry or dismiss them.
kernable_record_headgaugeThe newest record of the record log a standby follows (--replication sync).
kernable_standby_acknowledgedgaugeThe newest record the standby holds on its disk; writes after it wait.
kernable_standby_in_syncgauge1 while a standby is connected and has caught up; without it, writes wait.
kernable_standby_catching_upgauge1 while no standby has caught up yet (without a witness): writes are told without one.
kernable_store_bytesgaugeBytes the event store, the vault and the files take on the disk, measured every 10 s.
kernable_store_bytes_maxgauge--max-store-bytes: at it, new commands and file puts are refused (quota_storage).
kernable_readygauge1 while the kernel is ready: every aggregate that counts has a worker, every listener that counts is connected and caught up, and it is not shutting down.
kernable_standby_appliedgaugeOn a standby: the newest record of the primary's log it holds.
kernable_standby_laggaugeOn a standby: records it is behind the primary's head as last seen.
kernable_process_casesgaugeprocess, stateCases per process and state: running (a command sent or a query asked), waiting (for an event or a time), incident (stopped until an operator resolves it) held now; ended since the kernel started.
kernable_process_incidentscounterprocessIncidents raised since the kernel started, per process.
kernable_process_step_secondshistogramprocess, stepSeconds from reaching a step until it completed, per process and step.

kernable node ​

With metrics-listen in its [node] section, kernable node serves GET /metrics too, a line per child (the kernel, the backup follower, each service), named by the child label (Run a node from one file).

MetricTypeLabelsDescription
kernable_node_child_upgaugechildWhether the child runs (1) or not (0).
kernable_node_child_restarts_totalcounterchildTimes the node started the child again.
kernable_node_child_uptime_secondsgaugechildSeconds since the child last started, while it runs.
kernable_node_container_cpu_seconds_totalcounterchildCPU time a child's container used (a service, and on Linux the kernel and the backup follower), from its cgroup.
kernable_node_container_memory_bytesgaugechildMemory a child's container uses now, from its cgroup.