Appearance
Metrics and health
With --metrics-listen 127.0.0.1:9233 the kernel serves plain HTTP with no login on that address, so bind it to a private one. How to use it for health checks, alerts and graphs: Monitor kernable.
GET /metrics: Prometheus text format, below.GET /health: 200 while the process runs.GET /ready: 200 once every aggregate that counts has a worker connected and every listener that counts is connected and has caught up (see Who counts), else 503 naming the workers or listeners it waits for.
Counters and durations count since the kernel started and begin at 0 again after a restart. Durations are summaries without quantiles, in seconds: <name>_sum and <name>_count. The outcome label takes the values named in the description; the listener label is a listener's name, for every registered listener, connected or not.
| Metric | Type | Labels | Description |
|---|---|---|---|
kernable_commands_total | counter | outcome | Commands completed since the kernel started, by outcome: committed, or rejected by a worker or the kernel (scheduled commands cancelled, expired, or refused by a middleware when due included). Commands a middleware refused before the queue are not counted. |
kernable_queries_total | counter | outcome | Queries since the kernel started, by outcome: answered; rejected by a read model or a middleware; failed when no read model answered, because none was connected or none answered in time. |
kernable_middleware_total | counter | outcome, reason | Runs of a middleware chain, before or after a command or query, by outcome: passed, rejected, or failed when a middleware in the chain was not connected or did not answer within --middleware-timeout-ms. |
kernable_refused_total | counter | Requests and connections a limit refused since the kernel started, by reason: connections, requests_per_key, subscriptions_per_key, listeners_per_key, queued_commands, queued_per_aggregate, queued_bytes, cache_bytes, handshake_timeout, too_large, http_host, http_connections. | |
kernable_queued_commands | gauge | Commands stored in the queue and not completed yet: waiting for a worker or running. | |
kernable_scheduled_commands | gauge | Scheduled commands (deliver_at) not due yet, or due and being checked before they join their aggregate's queue; counted in kernable_queued_commands too. | |
kernable_worker_sessions | gauge | Worker sessions connected now. | |
kernable_head_position | gauge | Global position of the newest stored event; 0 while the store is empty. | |
kernable_execution_seconds | summary | Seconds from handing a command to a worker until its events are committed: the worker round trip plus storage. Committed commands only. | |
kernable_settle_seconds | summary | Seconds from the commit until every listener that counts acknowledged the events. Settling submits that settled in time only. | |
kernable_query_seconds | summary | Seconds from handing a query to the read models until one answered or rejected it. Queries that failed are not included. | |
kernable_middleware_seconds | summary | Seconds one middleware chain took to pass, reject or fail a request. | |
kernable_listener_position | gauge | listener | Global position of the last event the listener acknowledged. |
kernable_listener_lag | gauge | listener | Global positions between the newest stored event and the listener's position: how far it is behind. |
kernable_listener_connected | gauge | listener | 1 while the listener is connected, 0 while it is registered but away. |
kernable_listener_parked | gauge | Events the listener gave up on and parked, waiting for an operator to retry or dismiss them. | |
kernable_record_head | gauge | The newest record of the record log a standby follows (--replication sync). | |
kernable_standby_acknowledged | gauge | The newest record the standby holds on its disk; writes after it wait. | |
kernable_standby_in_sync | gauge | 1 while a standby is connected and has caught up; without it, writes wait. | |
kernable_standby_catching_up | gauge | 1 while no standby has caught up yet (without a witness): writes are told without one. | |
kernable_store_bytes | gauge | Bytes the event store, the vault and the files take on the disk, measured every 10 s. | |
kernable_store_bytes_max | gauge | --max-store-bytes: at it, new commands and file puts are refused (quota_storage). | |
kernable_ready | gauge | 1 while the kernel is ready: every aggregate that counts has a worker, every listener that counts is connected and caught up, and it is not shutting down. | |
kernable_standby_applied | gauge | On a standby: the newest record of the primary's log it holds. | |
kernable_standby_lag | gauge | On a standby: records it is behind the primary's head as last seen. | |
kernable_process_cases | gauge | process, state | Cases per process and state: running (a command sent or a query asked), waiting (for an event or a time), incident (stopped until an operator resolves it) held now; ended since the kernel started. |
kernable_process_incidents | counter | process | Incidents raised since the kernel started, per process. |
kernable_process_step_seconds | histogram | process, step | Seconds from reaching a step until it completed, per process and step. |
kernable node
With metrics-listen in its [node] section, kernable node serves GET /metrics too, a line per child (the kernel, the backup follower, each service), named by the child label (Run a node from one file).
| Metric | Type | Labels | Description |
|---|---|---|---|
kernable_node_child_up | gauge | child | Whether the child runs (1) or not (0). |
kernable_node_child_restarts_total | counter | child | Times the node started the child again. |
kernable_node_child_uptime_seconds | gauge | child | Seconds since the child last started, while it runs. |
kernable_node_container_cpu_seconds_total | counter | child | CPU time a child's container used (a service, and on Linux the kernel and the backup follower), from its cgroup. |
kernable_node_container_memory_bytes | gauge | child | Memory a child's container uses now, from its cgroup. |
