Servers
Any server replica serves any request, so servers scale freely. Enable the HorizontalPodAutoscaler and give the server a CPU request, which the target is a percent of:server.podDisruptionBudget.minAvailable (1).
A server holds the sync calls it accepted in memory until their results arrive. On shutdown it stops taking requests and gives open ones up to 30 seconds to finish. A sync call still waiting after that is cut off. The function still runs on its host, and its record still reaches the database.
Hosts
Withhost.autoscaling.enabled, a KEDA ScaledObject sizes the host Deployment instead of host.replicas. KEDA must already be installed in the cluster.
The demand signal
KEDA’s metrics-api scaler readsGET /scaling on the server’s Service every pollingInterval seconds:
Every server gives the same answer, since the queue’s depth is read from NATS. The endpoint is not authenticated, like
/healthz, so KEDA needs no credentials. Keep the server’s Service off the public internet, or block /scaling at the ingress.
The math
KEDA sets hosts todemand divided by the busy slots wanted per host, between minReplicas and maxReplicas:
targetUtilization to leave more headroom for bursts.
Scaling up and down
Hosts are added as soon as demand appears, with no stabilization window. They are removed one per minute once demand has stayed low forscaleDownStabilizationSeconds. A host that is told to stop takes no new invocations, releases its warm instances and gives in-flight invocations up to 40 seconds to finish before it cancels them. host.terminationGracePeriodSeconds (60) covers that. A canceled invocation retries if the function or call allows it.
Keep minReplicas at 1 or more. A new KVM node takes minutes to come up, and with no host at all every call waits for it.
KVM nodes
A new host only runs once it has a KVM node, and each host needs a node of its own. Run a node autoscaler such as Karpenter or Cluster Autoscaler on the KVM node pool, or pending hosts wait for a node indefinitely. The chart runs hosts as a Deployment rather than a DaemonSet for this reason: a pending host pod is what asks the node autoscaler for another node. Give the pool thealchemy.getgalaxy.io/kvm=true label and taint, an instance type with KVM, and a maximum size at least host.autoscaling.maxReplicas. See KVM nodes.