Previewkernable is not released yet: these docs describe work in progress, and things may change before 1.0.Not released yet.Request early access →
Skip to content

Errors ​

A request that does not succeed ends in one of two ways:

  • Rejected: an answer with a code and a message. A command handler, a query handler or a middleware decided against the request, or the kernel rejected a worker's result. This is an outcome, not a failure of the call.
  • Refused: a gRPC status instead of an answer. The kernel did not accept the request or could not complete it, or it could not be reached.

How the SDKs report them:

TypeScriptPythonRust
rejected commandsubmit returns { committed: false, code, message }submit returns Outcome(committed=False, code, message)submit returns Outcome::Rejected
rejected queryquery throws Rejectedquery raises Rejectedquery returns Err(Error::Rejected)
refusedthrows the gRPC error (code, details)raises grpc.RpcErrorErr(Error::Status)
committed, not settledthrows NotSettledraises NotSettledErr(Error::NotSettled)

A rejected command is final. A command rejected by its worker or by the kernel is completed and stored as rejected, and resubmitting it with the same command ID returns the same rejection. To try again, submit it with a new command ID. A command that a middleware refuses before it runs is not stored, and a rejected query is never stored.

Handlers choose their own codes: Rejected(message, code) in TypeScript and Python, Rejected::new(message).code(code) in Rust. The default code is rejected. Codes in the examples, such as not_found, forbidden or checked_out, are your application's; kernable gives them no meaning. The codes below come from kernable itself; make check fails if a code in the sources is missing from this page.

Rejection codes ​

CodeFromMeaningWhat to do
rejectedSDKThe default code of Rejected when a handler gives none.Read the message.
handler_errorSDK: worker, read modelA command or query handler threw something other than Rejected. For a command the message is only the error's type (TypeError; Rust: handler failed), as the rejection is kept for good; the worker logs the whole error. The command is not retried: a deterministic handler would throw again.Fix the handler, then submit again with a new command ID.
unknown_commandSDK: workerThe worker that got the command has no handler for its type, although the aggregate is registered for it: usually workers of different versions run side by side.Deploy workers that handle the type, then submit again with a new command ID.
snapshot_staleSDK: workerThe worker cannot read the snapshot it got (another schema). The kernel drops the snapshot and runs the command again with all its events, so a client never gets this code for a command that ran with a snapshot.Nothing.
version_mismatchkernelThe command's expected_version (request context) is not the aggregate's version: the aggregate changed since the client read it.Read the current state, decide again, and submit with a new command ID.
concurrency_conflictkernelThe aggregate kept changing while the command ran, also after five retries.Submit again with a new command ID.
invalid_event_typekernelA worker returned an event type that is empty or contains a dot.A bug in the worker or its SDK: fix it.
invalid_event_schemakernelA worker wrote an event at another schema version than it declared when it registered.A bug in the worker or its SDK: fix it.
invalid_event_originkernelA worker's event names where it came from (origin_partition, origin_position); only the kernel sets that, when a tenant moves to another partition.A bug in the worker or its SDK: fix it.
no_vaultkernelA worker's events refer to the vault, but the kernel has none.Start the kernel with --vault-key.
invalid_vault_valueskernelA worker's vault values do not match the references in its signed events, or could not be stored.A bug in the worker or its SDK; check the kernel log.
vault_errorkernelThe vault could not erase a subject.Check the kernel log and the vault store, then submit again with a new command ID.
key_revokedkernelThe command waited in the queue and was signed by a key revoked from a time at or before its arrival, or, for a scheduled command, before its time.If the command is legitimate, send it again signed with the sender's new key.
expiredkernelA scheduled command could not run by its until time, after an outage, say.Decide whether it should still happen; if so, send it again.
cancelledkernelA scheduled command was cancelled before it was due (Kernable.cancel); its ID keeps this result.Nothing: it was meant to stop.
too_latekernelA cancel came for a scheduled command that is due already, running or completed.Look at its result by its ID; undo it with another command if needed.
stalekernelA process's timer fell due after the step it belonged to was over (the awaited event came first). Shown among the rejections; the case is not affected.Nothing.
abandonedkernelA process engine's write was cut short by a crash; at the next start it is closed with this code and the engine writes again under a new ID.Nothing.
no_aggregateprocess incidentA command step's aggregate names a field of the case that is not there, so the command has no aggregate to go to.Fix the model or the data that fills the field; kernable processes skip or cancel the case.
no_timeprocess incidentA timer step's at names a field of the case that holds no number of ms.Fix the model or the data that fills the field; kernable processes skip or cancel the case.
no_pathprocess incidentA query step's answer matched none of its paths, and it has no otherwise.Add the path or an otherwise in a new version; kernable processes skip or cancel the case.
query_failedprocess incidentA query step's query could not be asked: nobody answered it for 30 s (the step asks again every second meanwhile, so a read model that is reconnecting does not stop the case), or the kernel could not reach who does. (A query its read model rejects stops the case with that rejection's code.)Start what answers it; then kernable processes retry the case.
loopprocess incidentA case took 200 steps without waiting: its decisions lead back to each other.Fix the conditions in a new version; cancel the case.
command_id_takenprocess incidentA command ID the engine fixed for a step is held by a command that is not the engine's. The case stops as an incident.Find what submitted that ID; then kernable processes retry the case.
not_foundkernelA cancel names a command the kernel does not know in that tenant. (Your handlers may use the same code for their own cases.)Check the command ID and the tenant.
too_largekernelThe command with its aggregate's snapshot, or one event of its history, is a larger message than --max-message-bytes, so no worker could receive it. (A longer history reaches the worker in parts.)Raise --max-message-bytes in the kernel and the SDKs' message limit, then submit again with a new command ID.
unique_takenkernelAnother aggregate holds a unique key the command would leave its aggregate with: funds.fund key isin is taken. The message names the key, never the value; so does the rejection's key field. Also for an aggregate whose key another one held before the key was declared, on every command until its value changes or becomes null.Choose another value, or free it on the aggregate that holds it; then submit with a new command ID (Unique keys and conditions).
requirement_missingkernelAn aggregate the command requires does not hold the fact when the command commits (funds.fund f1 does not hold open; named by a key: funds.fund with key isin does not hold open, never the value), or holds a fact it must not (funds.vehicle v1 holds woundUp). The rejection's requirement field is that requirement: the first of the command's that does not hold.Bring the other aggregate into that state first, or decide differently; then submit with a new command ID.
invalid_indexkernelA worker's unique keys, facts or requirements break a limit (64 keys, 64 facts, 64 requirements, names of at most 128 bytes without dots) or are malformed; the message says which.A bug in the aggregate's definition or its SDK: fix it.
erase_not_allowedkernelA worker returned an erasing event, but its key's grant has no vault_erase for that event type.Grant the worker's key vault_erase for the event type (Erase a person).
middleware_errorSDK: middlewareA middleware threw something other than Rejected. The request is refused, as by any rejection.Fix the middleware.
personal_in_clearSDK: workerAn event the handler returned holds, in clear, a field the command declares personal, but the event type does not declare it personal. It would have been written into the signed log for good, so the command is rejected instead.Declare the field personal on the event too (Domain.event(type, …).personal).
refusedSDK (Python): middlewareA Python middleware returned False. A return value other than None, True or False refuses too, with middleware_error.Check the request; the middleware decided against it.
not_a_middlewareSDK: listenerA check reached a listener that has no middleware handler for that point (before or after).Deploy the listener with the middleware it registered.
unknown_querySDK: read modelThe read model has no handler for the query type it registered to answer.Deploy a read model that answers the type.

gRPC statuses ​

StatusWhenWhat to do
UNAUTHENTICATEDA missing or invalid signature: on a command, query, subscription, result, rejection, grant, membership or revocation. A failed key challenge when a worker or listener connects. An expired request: queries, subscriptions, data and cache requests, memberships and revocations are valid for 60 s around their issued_at.Check the key and the clock.
PERMISSION_DENIEDThe key has no grant, its grant does not permit this action on this type (submit, run, subscribe, listen, query, answer, check, privileged, data_read, data_write, cache, manage, …), or the key is revoked. A grant not signed by a root key or signed by a revoked one, a grant in kernable-grant for another key than the one that signed the request, a revocation not signed by a root key. A worker or listener that connects with another key than the one its aggregate type or listener name is registered with, unless the new key's grant succeeds that key. A middleware that its service's membership does not list. Also ends a worker's, listener's or subscriber's stream when a newer grant or a revocation takes its rights away. Also: a file or cache request a middleware refused; the message is <code>: <message>, as the middleware gave it.Grant the key what it needs (Manage keys); for a middleware's refusal, see its code.
NOT_FOUNDA command for an aggregate type that no worker registered and no membership lists, or for a command type its workers did not register: often a typo. Commands of a registered or listed aggregate whose workers are away wait in the queue instead.Check the names; start the worker or apply the membership first.
ALREADY_EXISTSA listener connects under a name that a connected listener already uses (without an instance, or next to one without an instance), an instance connects whose ID is connected already, or a live listener uses the name of a registered one, or a registered one the name of a live one. A data or cache write (put, set, delete) sent again with the same signature: a replay, refused within its 60 s.Run one process per listener name, or give each process an instance (KERNABLE_INSTANCE); give a live listener another name. Sign every write anew; a replay needs no action.
RESOURCE_EXHAUSTEDA limit of the kernel is reached (Limits): the command queue is full (in total, for the aggregate, or in bytes; a command that is already stored is still answered), the key has too many requests in progress, open subscriptions or registered listeners, a listener has 64 instances connected, the key started more requests in the last second than --max-requests-per-second-per-key allows (rate), the stores reached --max-store-bytes (quota_storage: new commands and file puts only; queries, file deletes and the commands of --quota-exempt go on), or a cache value is larger than the whole cache. Nothing was stored. kernable_refused_total counts each reason.Retry later with backoff; if it persists, find the cause (a stopped worker lets the queue grow) or raise the limit. Remove unused listeners with kernable listeners remove.
INVALID_ARGUMENTA malformed request: a name that is not a valid domain or qualified type, a listener name or instance ID that is not 1 to 128 letters, digits, -, _ and ., a single listener without an instance or that is live, an instance of a replicated listener that parks an event, a worker ID over 256 bytes or with control characters, a missing command, vault values that do not match their references, an unknown listener name in a request from a listener's handler, a negative listener or middleware order, a data key not 1 to 512 bytes, a file over 16 MiB or a cache value over 1 MiB, a kernable-grant entry that is not a hex-encoded Grant, a malformed grant (an unknown action or type pattern), a malformed membership or worker registration.Fix the request; these are bugs in the caller.
FAILED_PRECONDITIONA protocol version the kernel does not speak (the message names which side to update). An outdated worker: it would write an older event version than the store holds. A request from a listener's handler that would wait for that listener itself. A switchover asked of a kernel that is not the primary of a pair, whose standby is not in sync, or whose standby did not hold every record within 10 s. An instance of a listener that is single while the connected ones are replicated, or the other way round (disagree on single). A request with vault values while the kernel has no vault (--vault-key). A membership issued no later than the one its service has: a replay, or the slower of two racing deploys. A data or cache write signed before the kernel started: it may have run already.Update the side the message names; route the request differently; start the kernel with a vault; apply the membership again; sign the write again.
UNAVAILABLEThe kernel cannot be reached, or the connection fails, also in the TLS handshake (for example a client certificate not signed by --tls-client-ca). A middleware that checks the request is not connected. No read model answers the query, or it went away while answering. The kernel is shutting down.Retry later. Workers and listeners of the SDKs reconnect by themselves, and SDK requests wait for the connection up to their timeout (60 s by default).
DEADLINE_EXCEEDEDA submit did not complete within --submit-timeout-ms: the command is stored and still runs. A query was not answered within --query-timeout-ms, or no read model caught up to its min_position. A middleware did not answer within --middleware-timeout-ms. A worker or listener stream did not complete its handshake within --handshake-timeout-ms. In the SDKs also: no connection to the kernel, or no answer, within the request's timeout (60 s by default; timeoutMs, timeout_s, Options::timeout).Resubmit the command with the same command ID to get its result. Retry a query; look for slow or stuck listeners (Monitor kernable).
INTERNALThe kernel's store failed, or a read model or middleware sent the wrong kind of reply.Check the kernel log.
NotSettled (SDK)Not a status: the command was committed, but not every listener that counts processed its events within --settle-timeout-ms. The outcome is attached.Treat the command as done; read models show it a little later.