Read an error
A call to/functions/<id>/invoke fails in one of two shapes.
The platform refused or could not run it. The status is the Connect code’s
HTTP status and the body names the code:
500, or 504 when it timed
out, and the body is the function’s error:
Alchemy-Invocation-Id when an invocation was created. Look it up
on the Invocations page or with GetInvocation for its status, attempts,
error and platformError, and read its stderr in the UI or with
GetInvocationLogs. On the API, Invoke returns a function’s failure in the
response’s error field and platform failures as Connect errors with the
same codes.
The version fails to build
CreateFunction returns while the version is BUILDING. A build that fails
turns the version FAILED with the reason in statusMessage and the tail of
the build output in buildLog:
A failed version stays failed. Fix the spec and create a new version. The
function keeps running its last ready version meanwhile.
Some specs are refused before anything builds.
CreateFunction and
CreateFunctionVersion answer invalid_argument or failed_precondition
with messages such as:
The function is not ready
The function has no
READY version yet, or the pinned one is still
building or failed. Wait for the build, or check its statusMessage.
The input is rejected
The host checks the input before it spends a microVM on it. Nothing runs, the invocation isREJECTED and it is never retried.
For an Arrow contract the violations read like
input is not an Arrow IPC stream: ..., input stream holds more than one record batch, field "id" has type int64, want utf8 or field "id" is nullable, but the contract does not allow nulls. Fields are compared by position, so a column in the wrong place
is reported as a wrong name.
The output is rejected
The function ran but its output broke the contract. The call answers500
with kind FUNCTION_ERROR_KIND_OUTPUT_INVALID, and it is not retried because
it would fail the same way.
Write diagnostics to stderr. Everything on stdout is the output.
The function crashes
A process that exits non-zero answers500 with kind
FUNCTION_ERROR_KIND_CRASHED. The message holds the exit status and the last
4 KiB of stderr:
A crash is retried up to the function’s
retries, 2 by default, after about
1 s, then 5 s, then 25 s, up to 5 minutes. The response’s Alchemy-Attempts
says how many runs the result took, and each attempt keeps its own log.
Out of memory
Alchemy has no error of its own for memory. When the function outgrows the microVM, the guest kernel kills it and the call reports a crash with status-1. If the whole guest stops answering instead, the call ends as a timeout
or an aborted platform error (409). Files in /tmp and /run live in memory and
count against the same limit.
Raise the function’s memory with UpdateFunction, which applies to the next
invocation of every version:
UpdateFunction sets name and description on every call, so send the
current ones. Send both resource fields: a zero one takes the server default
again. Memory defaults to 128 MiB and goes up to 8192 MiB.
The function times out
A function that runs past itstimeoutMs, 30 s by default, is killed with
everything it started. The call answers 504 with kind
FUNCTION_ERROR_KIND_TIMEOUT:
function exceeded its <n>ms timeout and the instance stopped responding. A timeout is retried
like a crash. Raise timeoutMs with UpdateFunction, up to 5 minutes, or
make the work smaller per call.
The call waits, then fails with a deadline
504 with code deadline_exceeded, and over RPC
deadline_exceeded. Every call waits in a
queue until a host has a free slot, and the call’s deadline covers that wait,
the build on a new host and every attempt.
Common causes:
- No hosts take work. Calls queue even with no hosts running. Check the Hosts page and Hosts do not start.
- Every slot is busy. Each host runs up to
MAX_INSTANCESinvocations at once, 8 by default, within itsMEMORY_BUDGET_MIB. A host without room hands the invocation back to the queue for another host. Add hosts. - The function is at
max_concurrency. Invocations past the limit wait in the queue,PENDING, until a running one finishes. There is no error for this. Raise or clear the limit withUpdateFunction(maxConcurrency: 0means no limit). - Retries filled the deadline. A retry runs only while the deadline leaves room for its backoff and another full timeout.
Prefer: respond-async) waits up to ASYNC_MAX_AGE, 24 hours
by default. One that no host took in time ends CANCELED with
expired in the queue before it could run. Follow an async call by
reading the Location header, /invocations/<id>, or with GetInvocation.
Image pull failures
Hosts pull images themselves. A pull that fails turns the versionFAILED
with a statusMessage such as:
An image with no build for the hosts’ architecture also fails here. Hosts
resolve
linux and their own CPU architecture.
A version that built on one host can still fail to build on another, such as
one without the same registry credentials. Its invocations there fail with
version <n> can no longer be built: <reason>; create a new version. Give
every host the same access, then create a new version.
Secrets are not available
A version whosespec.secrets names a secret a host cannot fetch fails each
attempt with unavailable:
503 once retries run out.
Check that the secret exists for the tenant and that the server has a secret
provider. See Secrets.
Hosts do not start
A host checks its machine before it joins. On failure it logs the reason and exits with status 1:
On Kubernetes, a host pod that stays
Pending usually has no node with the
alchemy.getgalaxy.io/kvm=true label. See
KVM nodes.
Idempotency keys
See HTTP invoke for how keys work.