Skip to main content
Every failure Alchemy reports carries a code and a message. This page lists the ones you are likely to meet, where they show up and what to change. The messages are quoted as the code writes them. Parts in angle brackets vary.

Read an error

A call to /functions/<id>/invoke fails in one of two shapes. The platform refused or could not run it. The status is the Connect code’s HTTP status and the body names the code:
The function ran and failed. The status is 500, or 504 when it timed out, and the body is the function’s error:
Both carry Alchemy-Invocation-Id when an invocation was created. Look it up on the Invocations page or with GetInvocation for its status, attempts, error and platformError, and read its stderr in the UI or with GetInvocationLogs. On the API, Invoke returns a function’s failure in the response’s error field and platform failures as Connect errors with the same codes.

The version fails to build

CreateFunction returns while the version is BUILDING. A build that fails turns the version FAILED with the reason in statusMessage and the tail of the build output in buildLog:
A failed version stays failed. Fix the spec and create a new version. The function keeps running its last ready version meanwhile. Some specs are refused before anything builds. CreateFunction and CreateFunctionVersion answer invalid_argument or failed_precondition with messages such as:

The function is not ready

The function has no READY version yet, or the pinned one is still building or failed. Wait for the build, or check its statusMessage.

The input is rejected

The host checks the input before it spends a microVM on it. Nothing runs, the invocation is REJECTED and it is never retried. For an Arrow contract the violations read like input is not an Arrow IPC stream: ..., input stream holds more than one record batch, field "id" has type int64, want utf8 or field "id" is nullable, but the contract does not allow nulls. Fields are compared by position, so a column in the wrong place is reported as a wrong name.

The output is rejected

The function ran but its output broke the contract. The call answers 500 with kind FUNCTION_ERROR_KIND_OUTPUT_INVALID, and it is not retried because it would fail the same way. Write diagnostics to stderr. Everything on stdout is the output.

The function crashes

A process that exits non-zero answers 500 with kind FUNCTION_ERROR_KIND_CRASHED. The message holds the exit status and the last 4 KiB of stderr:
A crash is retried up to the function’s retries, 2 by default, after about 1 s, then 5 s, then 25 s, up to 5 minutes. The response’s Alchemy-Attempts says how many runs the result took, and each attempt keeps its own log.

Out of memory

Alchemy has no error of its own for memory. When the function outgrows the microVM, the guest kernel kills it and the call reports a crash with status -1. If the whole guest stops answering instead, the call ends as a timeout or an aborted platform error (409). Files in /tmp and /run live in memory and count against the same limit. Raise the function’s memory with UpdateFunction, which applies to the next invocation of every version:
UpdateFunction sets name and description on every call, so send the current ones. Send both resource fields: a zero one takes the server default again. Memory defaults to 128 MiB and goes up to 8192 MiB.

The function times out

A function that runs past its timeoutMs, 30 s by default, is killed with everything it started. The call answers 504 with kind FUNCTION_ERROR_KIND_TIMEOUT:
When the microVM itself stops answering, the message is function exceeded its <n>ms timeout and the instance stopped responding. A timeout is retried like a crash. Raise timeoutMs with UpdateFunction, up to 5 minutes, or make the work smaller per call.

The call waits, then fails with a deadline

The function never returned in time, usually because it never started. Over plain HTTP this answers 504 with code deadline_exceeded, and over RPC deadline_exceeded. Every call waits in a queue until a host has a free slot, and the call’s deadline covers that wait, the build on a new host and every attempt. Common causes:
  • No hosts take work. Calls queue even with no hosts running. Check the Hosts page and Hosts do not start.
  • Every slot is busy. Each host runs up to MAX_INSTANCES invocations at once, 8 by default, within its MEMORY_BUDGET_MIB. A host without room hands the invocation back to the queue for another host. Add hosts.
  • The function is at max_concurrency. Invocations past the limit wait in the queue, PENDING, until a running one finishes. There is no error for this. Raise or clear the limit with UpdateFunction (maxConcurrency: 0 means no limit).
  • Retries filled the deadline. A retry runs only while the deadline leaves room for its backoff and another full timeout.
An async call (Prefer: respond-async) waits up to ASYNC_MAX_AGE, 24 hours by default. One that no host took in time ends CANCELED with expired in the queue before it could run. Follow an async call by reading the Location header, /invocations/<id>, or with GetInvocation.

Image pull failures

Hosts pull images themselves. A pull that fails turns the version FAILED with a statusMessage such as:
An image with no build for the hosts’ architecture also fails here. Hosts resolve linux and their own CPU architecture. A version that built on one host can still fail to build on another, such as one without the same registry credentials. Its invocations there fail with version <n> can no longer be built: <reason>; create a new version. Give every host the same access, then create a new version.

Secrets are not available

A version whose spec.secrets names a secret a host cannot fetch fails each attempt with unavailable:
The attempt is retried, and the call answers 503 once retries run out. Check that the secret exists for the tenant and that the server has a secret provider. See Secrets.

Hosts do not start

A host checks its machine before it joins. On failure it logs the reason and exits with status 1: On Kubernetes, a host pod that stays Pending usually has no node with the alchemy.getgalaxy.io/kvm=true label. See KVM nodes.

Idempotency keys

See HTTP invoke for how keys work.