Appearance
Errors
A request that does not succeed ends in one of two ways:
- Rejected: an answer with a code and a message. A command handler, a query handler or a middleware decided against the request, or the kernel rejected a worker's result. This is an outcome, not a failure of the call.
- Refused: a gRPC status instead of an answer. The kernel did not accept the request or could not complete it, or it could not be reached.
How the SDKs report them:
| TypeScript | Python | Rust | |
|---|---|---|---|
| rejected command | submit returns { committed: false, code, message } | submit returns Outcome(committed=False, code, message) | submit returns Outcome::Rejected |
| rejected query | query throws Rejected | query raises Rejected | query returns Err(Error::Rejected) |
| refused | throws the gRPC error (code, details) | raises grpc.RpcError | Err(Error::Status) |
| committed, not settled | throws NotSettled | raises NotSettled | Err(Error::NotSettled) |
A rejected command is final. A command rejected by its worker or by the kernel is completed and stored as rejected, and resubmitting it with the same command ID returns the same rejection. To try again, submit it with a new command ID. A command that a middleware refuses before it runs is not stored, and a rejected query is never stored.
Handlers choose their own codes: Rejected(message, code) in TypeScript and Python, Rejected::new(message).code(code) in Rust. The default code is rejected. Codes in the examples, such as not_found, forbidden or checked_out, are your application's; kernable gives them no meaning. The codes below come from kernable itself; make check fails if a code in the sources is missing from this page.
Rejection codes
| Code | From | Meaning | What to do |
|---|---|---|---|
rejected | SDK | The default code of Rejected when a handler gives none. | Read the message. |
handler_error | SDK: worker, read model | A command or query handler threw something other than Rejected. For a command the message is only the error's type (TypeError; Rust: handler failed), as the rejection is kept for good; the worker logs the whole error. The command is not retried: a deterministic handler would throw again. | Fix the handler, then submit again with a new command ID. |
unknown_command | SDK: worker | The worker that got the command has no handler for its type, although the aggregate is registered for it: usually workers of different versions run side by side. | Deploy workers that handle the type, then submit again with a new command ID. |
snapshot_stale | SDK: worker | The worker cannot read the snapshot it got (another schema). The kernel drops the snapshot and runs the command again with all its events, so a client never gets this code for a command that ran with a snapshot. | Nothing. |
version_mismatch | kernel | The command's expected_version (request context) is not the aggregate's version: the aggregate changed since the client read it. | Read the current state, decide again, and submit with a new command ID. |
concurrency_conflict | kernel | The aggregate kept changing while the command ran, also after five retries. | Submit again with a new command ID. |
invalid_event_type | kernel | A worker returned an event type that is empty or contains a dot. | A bug in the worker or its SDK: fix it. |
invalid_event_schema | kernel | A worker wrote an event at another schema version than it declared when it registered. | A bug in the worker or its SDK: fix it. |
invalid_event_origin | kernel | A worker's event names where it came from (origin_partition, origin_position); only the kernel sets that, when a tenant moves to another partition. | A bug in the worker or its SDK: fix it. |
no_vault | kernel | A worker's events refer to the vault, but the kernel has none. | Start the kernel with --vault-key. |
invalid_vault_values | kernel | A worker's vault values do not match the references in its signed events, or could not be stored. | A bug in the worker or its SDK; check the kernel log. |
vault_error | kernel | The vault could not erase a subject. | Check the kernel log and the vault store, then submit again with a new command ID. |
key_revoked | kernel | The command waited in the queue and was signed by a key revoked from a time at or before its arrival, or, for a scheduled command, before its time. | If the command is legitimate, send it again signed with the sender's new key. |
expired | kernel | A scheduled command could not run by its until time, after an outage, say. | Decide whether it should still happen; if so, send it again. |
cancelled | kernel | A scheduled command was cancelled before it was due (Kernable.cancel); its ID keeps this result. | Nothing: it was meant to stop. |
too_late | kernel | A cancel came for a scheduled command that is due already, running or completed. | Look at its result by its ID; undo it with another command if needed. |
stale | kernel | A process's timer fell due after the step it belonged to was over (the awaited event came first). Shown among the rejections; the case is not affected. | Nothing. |
abandoned | kernel | A process engine's write was cut short by a crash; at the next start it is closed with this code and the engine writes again under a new ID. | Nothing. |
no_aggregate | process incident | A command step's aggregate names a field of the case that is not there, so the command has no aggregate to go to. | Fix the model or the data that fills the field; kernable processes skip or cancel the case. |
no_time | process incident | A timer step's at names a field of the case that holds no number of ms. | Fix the model or the data that fills the field; kernable processes skip or cancel the case. |
no_path | process incident | A query step's answer matched none of its paths, and it has no otherwise. | Add the path or an otherwise in a new version; kernable processes skip or cancel the case. |
query_failed | process incident | A query step's query could not be asked: nobody answered it for 30 s (the step asks again every second meanwhile, so a read model that is reconnecting does not stop the case), or the kernel could not reach who does. (A query its read model rejects stops the case with that rejection's code.) | Start what answers it; then kernable processes retry the case. |
loop | process incident | A case took 200 steps without waiting: its decisions lead back to each other. | Fix the conditions in a new version; cancel the case. |
command_id_taken | process incident | A command ID the engine fixed for a step is held by a command that is not the engine's. The case stops as an incident. | Find what submitted that ID; then kernable processes retry the case. |
not_found | kernel | A cancel names a command the kernel does not know in that tenant. (Your handlers may use the same code for their own cases.) | Check the command ID and the tenant. |
too_large | kernel | The command with its aggregate's snapshot, or one event of its history, is a larger message than --max-message-bytes, so no worker could receive it. (A longer history reaches the worker in parts.) | Raise --max-message-bytes in the kernel and the SDKs' message limit, then submit again with a new command ID. |
unique_taken | kernel | Another aggregate holds a unique key the command would leave its aggregate with: funds.fund key isin is taken. The message names the key, never the value; so does the rejection's key field. Also for an aggregate whose key another one held before the key was declared, on every command until its value changes or becomes null. | Choose another value, or free it on the aggregate that holds it; then submit with a new command ID (Unique keys and conditions). |
requirement_missing | kernel | An aggregate the command requires does not hold the fact when the command commits (funds.fund f1 does not hold open; named by a key: funds.fund with key isin does not hold open, never the value), or holds a fact it must not (funds.vehicle v1 holds woundUp). The rejection's requirement field is that requirement: the first of the command's that does not hold. | Bring the other aggregate into that state first, or decide differently; then submit with a new command ID. |
invalid_index | kernel | A worker's unique keys, facts or requirements break a limit (64 keys, 64 facts, 64 requirements, names of at most 128 bytes without dots) or are malformed; the message says which. | A bug in the aggregate's definition or its SDK: fix it. |
erase_not_allowed | kernel | A worker returned an erasing event, but its key's grant has no vault_erase for that event type. | Grant the worker's key vault_erase for the event type (Erase a person). |
middleware_error | SDK: middleware | A middleware threw something other than Rejected. The request is refused, as by any rejection. | Fix the middleware. |
personal_in_clear | SDK: worker | An event the handler returned holds, in clear, a field the command declares personal, but the event type does not declare it personal. It would have been written into the signed log for good, so the command is rejected instead. | Declare the field personal on the event too (Domain.event(type, …).personal). |
refused | SDK (Python): middleware | A Python middleware returned False. A return value other than None, True or False refuses too, with middleware_error. | Check the request; the middleware decided against it. |
not_a_middleware | SDK: listener | A check reached a listener that has no middleware handler for that point (before or after). | Deploy the listener with the middleware it registered. |
unknown_query | SDK: read model | The read model has no handler for the query type it registered to answer. | Deploy a read model that answers the type. |
gRPC statuses
| Status | When | What to do |
|---|---|---|
UNAUTHENTICATED | A missing or invalid signature: on a command, query, subscription, result, rejection, grant, membership or revocation. A failed key challenge when a worker or listener connects. An expired request: queries, subscriptions, data and cache requests, memberships and revocations are valid for 60 s around their issued_at. | Check the key and the clock. |
PERMISSION_DENIED | The key has no grant, its grant does not permit this action on this type (submit, run, subscribe, listen, query, answer, check, privileged, data_read, data_write, cache, manage, …), or the key is revoked. A grant not signed by a root key or signed by a revoked one, a grant in kernable-grant for another key than the one that signed the request, a revocation not signed by a root key. A worker or listener that connects with another key than the one its aggregate type or listener name is registered with, unless the new key's grant succeeds that key. A middleware that its service's membership does not list. Also ends a worker's, listener's or subscriber's stream when a newer grant or a revocation takes its rights away. Also: a file or cache request a middleware refused; the message is <code>: <message>, as the middleware gave it. | Grant the key what it needs (Manage keys); for a middleware's refusal, see its code. |
NOT_FOUND | A command for an aggregate type that no worker registered and no membership lists, or for a command type its workers did not register: often a typo. Commands of a registered or listed aggregate whose workers are away wait in the queue instead. | Check the names; start the worker or apply the membership first. |
ALREADY_EXISTS | A listener connects under a name that a connected listener already uses (without an instance, or next to one without an instance), an instance connects whose ID is connected already, or a live listener uses the name of a registered one, or a registered one the name of a live one. A data or cache write (put, set, delete) sent again with the same signature: a replay, refused within its 60 s. | Run one process per listener name, or give each process an instance (KERNABLE_INSTANCE); give a live listener another name. Sign every write anew; a replay needs no action. |
RESOURCE_EXHAUSTED | A limit of the kernel is reached (Limits): the command queue is full (in total, for the aggregate, or in bytes; a command that is already stored is still answered), the key has too many requests in progress, open subscriptions or registered listeners, a listener has 64 instances connected, the key started more requests in the last second than --max-requests-per-second-per-key allows (rate), the stores reached --max-store-bytes (quota_storage: new commands and file puts only; queries, file deletes and the commands of --quota-exempt go on), or a cache value is larger than the whole cache. Nothing was stored. kernable_refused_total counts each reason. | Retry later with backoff; if it persists, find the cause (a stopped worker lets the queue grow) or raise the limit. Remove unused listeners with kernable listeners remove. |
INVALID_ARGUMENT | A malformed request: a name that is not a valid domain or qualified type, a listener name or instance ID that is not 1 to 128 letters, digits, -, _ and ., a single listener without an instance or that is live, an instance of a replicated listener that parks an event, a worker ID over 256 bytes or with control characters, a missing command, vault values that do not match their references, an unknown listener name in a request from a listener's handler, a negative listener or middleware order, a data key not 1 to 512 bytes, a file over 16 MiB or a cache value over 1 MiB, a kernable-grant entry that is not a hex-encoded Grant, a malformed grant (an unknown action or type pattern), a malformed membership or worker registration. | Fix the request; these are bugs in the caller. |
FAILED_PRECONDITION | A protocol version the kernel does not speak (the message names which side to update). An outdated worker: it would write an older event version than the store holds. A request from a listener's handler that would wait for that listener itself. A switchover asked of a kernel that is not the primary of a pair, whose standby is not in sync, or whose standby did not hold every record within 10 s. An instance of a listener that is single while the connected ones are replicated, or the other way round (disagree on single). A request with vault values while the kernel has no vault (--vault-key). A membership issued no later than the one its service has: a replay, or the slower of two racing deploys. A data or cache write signed before the kernel started: it may have run already. | Update the side the message names; route the request differently; start the kernel with a vault; apply the membership again; sign the write again. |
UNAVAILABLE | The kernel cannot be reached, or the connection fails, also in the TLS handshake (for example a client certificate not signed by --tls-client-ca). A middleware that checks the request is not connected. No read model answers the query, or it went away while answering. The kernel is shutting down. | Retry later. Workers and listeners of the SDKs reconnect by themselves, and SDK requests wait for the connection up to their timeout (60 s by default). |
DEADLINE_EXCEEDED | A submit did not complete within --submit-timeout-ms: the command is stored and still runs. A query was not answered within --query-timeout-ms, or no read model caught up to its min_position. A middleware did not answer within --middleware-timeout-ms. A worker or listener stream did not complete its handshake within --handshake-timeout-ms. In the SDKs also: no connection to the kernel, or no answer, within the request's timeout (60 s by default; timeoutMs, timeout_s, Options::timeout). | Resubmit the command with the same command ID to get its result. Retry a query; look for slow or stuck listeners (Monitor kernable). |
INTERNAL | The kernel's store failed, or a read model or middleware sent the wrong kind of reply. | Check the kernel log. |
NotSettled (SDK) | Not a status: the command was committed, but not every listener that counts processed its events within --settle-timeout-ms. The outcome is attached. | Treat the command as done; read models show it a little later. |
