The microVM
A microVM gets exactly what its function’sresources ask for: vcpus and
memory_mib. Its devices are:
- a read-only root drive, the function’s image flattened into ext4,
- a read-only layer drive, when the version has source files or a compiled layer,
- a vsock to its host, and a virtio-rng device for entropy,
- a network interface, only when the function sets
network: EGRESS.
EGRESS a guest has no network. Its one channel out is the vsock,
where the supervisor serves the host.
Egress
A function withnetwork: EGRESS reaches the internet and nothing else:
- Namespace: each microVM runs in a network namespace of its own. The
guest sits at
172.16.0.2behind a tap device, the same in every namespace, so snapshots restore unchanged. A veth pair with an address fromEGRESS_SUBNETlinks the namespace to the host. - Deny list: the host drops traffic from guests to private, shared,
loopback, link-local and reserved ranges, so cloud metadata at
169.254.169.254, other pods, NATS and PostgreSQL stay unreachable.EGRESS_DENYadds ranges, such as cluster ranges outside the private ones. - Host and peers: a guest cannot reach the host itself or another microVM.
- NAT: the rest leaves masqueraded as the host pod.
- DNS: the guest uses the resolvers in
EGRESS_DNS, 1.1.1.1 and 8.8.8.8 by default. They must be outside the denied ranges, so cluster DNS is not one of them. - IPv6: dropped.
ip or nft, starts anyway and
fails EGRESS invocations with the reason.
Inside, the supervisor is PID 1. It overlays the layer drive on the root
filesystem read-only, mounts /proc, /sys, and a tmpfs on /tmp and /run,
and runs the function as one process per invocation.
The jailer
Every microVM runs under Firecracker’s jailer, which the host drives as root:- chroot: Firecracker runs inside its own directory under
<DATA_DIR>/jail/firecracker/<id>/root, holding hard links to the kernel and the drives and nothing else. Its log stays outside the chroot. - cgroup: each microVM gets its own cgroup v2 under the host’s, with
cpu.maxset to its vCPU count, so it cannot take more CPU time than it asked for. - uid: each microVM runs as its own unprivileged uid and gid, taken from a
range of 65536 starting at
UID_BASE, 10000000 by default. No two running microVMs share one, so a process that escaped one guest cannot signal or read another.
Snapshot restore
After a version builds, the host boots it once, waits for the supervisor to start, pauses the microVM and writes a full snapshot of its memory and device state. Each new instance restores that snapshot instead of booting a kernel. The memory file is shared read-only between restores, and each restore maps it privately, so what one instance writes no other sees. A snapshot is only good for the Firecracker, kernel and supervisor it was taken with, so the cache is keyed by those three and older snapshots are dropped when a host starts. A restore that fails falls back to a cold boot.SNAPSHOTS_ENABLED
turns snapshots off.
Every instance restored from one snapshot wakes with the same kernel random
state and the clock of the moment it was taken. Before each invocation the
supervisor corrects the guest clock when it is more than a second off the
host’s, and reseeds the kernel’s random generator from the virtio-rng device, so
two instances never hand out the same random bytes.
Memory and vCPU budgeting
A host counts each instance at its full
memory_mib. When a new instance would
go over the budget, the host evicts warm instances of other artifacts, least
recently used first. When there is nothing left to evict, it hands the
invocation back to the queue for another host.
What is shared between invocations
A host keeps an instance warm after an invocation, forWARM_INSTANCE_TTL, 5
minutes by default, and the next invocation of the same artifact reuses it.
That makes the next call fast, and it decides what two invocations can share.
An instance belongs to one artifact, which is one version of one function in
one tenant, so a warm instance is only ever reused by invocations of that
version. It runs one invocation at a time.
Set
WARM_INSTANCE_TTL to 0s to release every instance after one invocation.
Every call then starts from a fresh snapshot restore, and nothing a function
writes reaches the next call.PREWARM_COUNT keeps that many idle instances booted per artifact before any
invocation arrives. They are booted from free memory only, never by evicting
other instances.