Skip to main content

ADR 0052: Kubernetes Is Both a Node Source and a Runtime; Cards Are Borrowed from the kubelet on Demand

On this page

Date: 2026-10-19. Status: Accepted.

Background

Users already have Kubernetes clusters with GPUs and want to connect them without installing an agent on each machine, while keeping gpuworks workloads visible and accountable inside Kubernetes. More than thirty places in the server depend on there being “an agent behind the node”; the agent handles probing, images, model files, health checks, sleep, logs, terminals, and tunnels.

Decisions

  1. The agent runs as a DaemonSet, with a new kube runtime. An instance is a bare Pod pinned to this node (spec.nodeName, restartPolicy: Never, host network, hostPath mounts at identical paths, and an ownerReference pointing to the agent's DaemonSet). The desired state the server delivers does not change, and the server does not know that the executing carrier is a Pod; this is isomorphic to the Mac host runtime (ADR 0044). There are no agentless virtual nodes and no Operator or CRD.
  2. A cluster connect token is exchanged for a node token. A dedicated POST /api/v1/kube/enroll is idempotent per (cluster, node name) and binds to the cluster on first use (the UID of the kube-system namespace); the hub handshake is unchanged. The server holds no kubeconfig and does not connect to the apiserver (amended on 2026-10-22 by ADR 0063: for clusters connected from a cloud account, after user confirmation the manifest is applied with a short-lived kubeconfig the cloud provider issues on the spot, and it is not stored).
  3. Each card has one allocator. When a device plugin exists, the cards belong to the kubelet, and gpuworks starts placeholder Pods that each request one card for the cards it needs. The device plugin does not accept a specified device, so when borrowing, one placeholder Pod is started for every idle card at once (the kubelet has no choice), the card each placeholder Pod received is identified through the kubelet PodResources, and the unneeded ones are returned. Cards are returned after the last instance has stopped for longer than the cluster's idle time (default 60 seconds, changed from 600 seconds on 2026-10-20; -1 keeps them permanently; the server pushes this time live, see the 2026-11-14 amendment). Cards allocated to other Pods are reported as externally occupied, following the existing rule that a card occupied outside the platform is not scheduled; the scheduler is unchanged.
  4. Only the standard library is used. The apiserver is accessed with a minimal REST client (in-cluster ServiceAccount), PodResources with the standard library's h2c and hand-written protobuf decoding; image operations are delegated to nerdctl in the k8s.io namespace. client-go and gRPC are not introduced.
  5. Devices go through the agent's own device plugin, and commands enter the node's mount namespace (added 2026-10-20). The device nodes an instance needs reuse the injection already validated on each vendor's docker / nerdctl path (the --device list and driver directory mounts). The agent registers with the kubelet one resource per card, gpuworks.ai/accelerator-<card number>, and a shared resource gpuworks.ai/accelerator-shared for nodes whose cards are shared; the instance Pod requests what it needs, and the kubelet hands the device nodes to the container through Allocate. No privileged mode is required, and neither the vendor container runtime nor the vendor device plugin is needed. Resource names are split per card because Allocate carries no Pod identity, so only the resource name can designate which card is meant. A Pod annotation plus CDI was tried: the kubelet does not pass Pod annotations to the container, and containerd recognizes only the CDI supplied by a device plugin, so on real machines the device was not injected (a PPU instance failed with Failed to infer device type). Through hostPID, the agent runs nerdctl, mount, and vendor CLIs in the node's mount namespace with nsenter -t 1 -m: the containerd socket, certs.d, the exec FIFO, NFS mounts, and the C libraries the vendor CLIs depend on are all on the node's own paths and versions (the PPU SDK needs glibc 2.38, while the agent image has 2.36). The NVIDIA CLI is injected by the NVIDIA runtime: when the cluster entry specifies a RuntimeClass, the agent runs under it; when not (the default), the agent container uses the node's default runtime, which on managed Kubernetes is NVIDIA's and injects nvidia-smi; when it is not injected (k3s defaults to runc), the agent runs it on the node. Instance Pods use the cluster's nvidia RuntimeClass, or the default runtime if there is none (the server now pushes the instance Pods' RuntimeClass live, see the 2026-11-14 amendment; these sentences on the agent's runtime and nvidia-smi are superseded by the 2026-11-12 amendment: the agent always runs under the default runtime, and the amendment gives the nvidia-smi lookup). The containerd the kubelet uses is determined from the -address of the agent container's shim, not guessed from the socket location. One cluster entry has one vendor (superseded by the 2026-11-12 amendment: an entry has no vendor). When the device plugin allocates less than a whole card (time-slicing, MIG, vNPU without HAMi's registration), cards are not borrowed, and the card is reported as externally occupied with the reason; HAMi and ACK GPU sharing are item 7.
  6. The lifecycle follows Kubernetes. When a node is cordoned, the platform cordons it too; a node offline for 15 minutes is removed once an online agent in the same cluster confirms it no longer exists and it has no instances; a node reports its own engine port range, which avoids the NodePort range (superseded by the 2026-11-14 amendment: the entry decides the engine port range and the server pushes it). Upgrades are item 8 (amended on 2026-11-12; before that an upgrade switched the DaemonSet to the image of the server's version).
  7. How cards are borrowed depends on what allocates them on the node (added 2026-11-10). HAMi and ACK GPU sharing do not hand out "one unit, one card", so whole cards are borrowed as described below; other splits are still refused with a stated reason (MIG, time-slicing, vNPU templates without HAMi's registration).
    • HAMi (the node has a hami.io/node-*-register annotation): the placeholder Pod sets schedulerName (hami-scheduler by default, overridable with the environment variable GPUWORKS_KUBE_HAMI_SCHEDULER) and nodeSelector kubernetes.io/hostname, and no nodeName (HAMi's device plugin does not serve a Pod that bypassed the scheduler); it requests one whole card and names it by UUID, so the card wanted is the card borrowed instead of borrowing everything and returning the rest. Who owns the card is what HAMi wrote in hami.io/vgpu-devices-allocated on the placeholder Pod, checked to be a whole card (the kubelet's device IDs are <UUID>-<share number> and cannot be parsed as card numbers).
    • ACK GPU sharing (the aliyun.com/gpu-mem resource, or the label ack.node.gpu.schedule is share, cgpu, or core_mem): the placeholder Pod requests the GiB of one card; there is no way to name a card, so the card borrowed is read from scheduler.framework.gpushare.allocation, keeping the existing "borrow every free card, keep the wanted ones, return the rest".
    • External occupancy comes from the allocation annotations on other Pods on the same node (so the agent reads the Pods of every namespace on its node); a placeholder Pod the scheduler leaves Pending fails the borrow, with the scheduler's reason and the Pods holding the card.
    • When the node changes mode while the platform holds cards, the new scheduler's books do not contain the old placeholder Pods and give the same card out again: those cards are reported as externally occupied with the reason, no more cards are borrowed in the old mode, and the old placeholder Pods are deleted once no instance uses their card.
    • HAMi 2.11 and ACK GPU sharing behavior for NVIDIA was checked on an Alibaba Cloud ACK cluster (Kubernetes 1.36); HAMi on Ascend follows HAMi's documented shape and has only been verified against fixtures, not on hardware.
  8. The image is only the agent's runtime; the agent runs and is upgraded in the node's state directory (added 2026-11-12).
    • The image's tag is the runtime's revision (build/agent-image/REVISION, such as r1), raised only when the base image or nerdctl changes; the manifest names that tag, whatever the server's version.
    • When the container starts, the image's agent copies itself to agent/gpuworks-agent in the state directory (a hostPath, which outlives the Pod) and execs it; later starts of the same image keep the copy there, and another image copies again. A fresh copy takes the server's own agent from the server's /dl/ once before it registers, the same version an installed node gets.
    • From then on an upgrade is an ordinary node's in-place upgrade (the binary is pushed or pulled, exec'd, confirmed once it registers, and rolled back to the previous one otherwise), with the container's restart taking systemd's place as the supervisor; the batch upgrade no longer leaves such nodes out (the agent declares the capability upgrade.kube_binary).
    • When the runtime revision changes, the manifest renders the new tag, and applying it once more moves the DaemonSet to the new image; the agent no longer has the right to change its own DaemonSet (the Role grants only get on it). With the state directory mounted noexec the agent keeps running the image's program and does not declare the capability; upgrades skip it and say why.
    • The image is published to Docker Hub, quay.io and Huawei Cloud SWR; the entry records which registry it pulls from (agent_image_source), preselected by region for a cluster of a cloud account (SWR in Alibaba Cloud's mainland China regions) and chosen by the user for one connected by hand; an image of the user's own takes precedence.
  9. The priority of the hold and instance Pods is the entry's, and they do not preempt by default (added 2026-11-12). The PriorityClass is named per namespace (gpuworks-inference-<namespace>, so several entries of one cluster do not overwrite each other), and the agent learns the name from the DaemonSet's environment; its value is 10000 by default (fixed at 100000 before) with preemptionPolicy: Never by default. With HAMi or ACK GPU sharing the hold Pods go through the scheduler, and when the platform's view of the cards and the scheduler's books disagree (a Pod scheduled between reading the allocations and creating the hold Pod, say), the former default let the scheduler evict lower-priority ordinary Pods; now the hold Pod only waits and the borrow fails. Kubernetes does not let a PriorityClass's value or preemption policy change: changing them means deleting and recreating it.

Trade-offs

  • Bare Pods instead of a Deployment. Retries, whole-group failure, and draining are already handled by unit reconciliation (ADR 0029); handing them to a Kubernetes controller as well would give two reconcile loops contending for the same Pods.
  • Host network. Port allocation, loopback health checks, tunnels, serving forwarding, and NCCL/RDMA NIC selection all work unchanged; the cost is that instances do not use the Pod network or Services.
  • Borrowing cards instead of disabling the device plugin. The user need not change an existing GPU Operator and the Kubernetes books stay accurate; the cost is that at the moment of borrowing all idle cards are briefly occupied (other Pods only go Pending briefly), and nodes with time-slicing / MIG replicas cannot be used.
  • A privileged agent. Probing, NFS mount propagation, entering the node's mount namespace, and registering the device plugin require it; the manifest labels the namespace with PSA privileged.
  • The binary in the state directory rather than in the image (item 8). An upgrade touches no registry and does not change the DaemonSet, private registries and offline clusters need not take an image for every version, and the entry point is the same as for any node; the cost is that the image tag kubectl shows does not tell the agent's version (the platform's does), and a node added to the cluster still pulls the runtime image once. Distributing each new image into every node's containerd over the agent connection and then changing the DaemonSet was considered: it also avoids registries, but sends an image to every node on each upgrade instead of a binary.
  • Reusing the vendor device plugin instead of shipping our own. Shipping our own device plugin for each vendor would require the user to turn the vendor's off; gpuworks is a guest in the cluster and keeps the plugin the user already installed. The cost is that cards can be borrowed only whole, and nodes the cluster itself has already split (MIG, time-slicing, and so on) cannot be used; once a whole card is borrowed, gpuworks' own dGPU slicing works on it as usual.

Consequences

  • New persisted facts: the kube_clusters table; kube_cluster_id, kube_node_name, and kube_unschedulable on nodes; and engine_port_range in the network inventory snapshot.
  • New packages: internal/server/kubemanifest and internal/agent/runtime/kube (including the two leaf packages kubeapi and podresources).
  • The agent image (build/agent-image, debian-slim: the nvidia-smi mounted in by the NVIDIA runtime and the host's npu-smi are both glibc programs; COPY only, so both amd64 and arm64 build without emulation). It is published as the runtime revision rN and as rN-<server version>.
  • New package: internal/agent/kubeboot (the hand-over and alignment of item 8); new persisted fact: agent_image_source on kube_clusters (nothing at render time tells which registry the nodes can reach).
  • New persisted facts: priority and allow_preemption on kube_clusters (item 9).
  • New persisted facts: vendor (removed by the 2026-11-12 amendment), state_dir, and api_server on kube_clusters (manifest parameters). The agent uses the node's DNS (dnsPolicy: Default) and does not depend on cluster DNS; nodes where no component provides Service addresses (KubeEdge and OpenYurt edge nodes) connect directly to the API server using api_server.

Amendment (2026-11-12): Zero Vendor Configuration

A cluster entry no longer has a vendor. Entries used to choose one, and the manifest mounted vendor-specific host paths into the agent accordingly (driver directories, vendor CLIs, device nodes); this amendment removes the field and has the agent detect it itself:

  1. An entry has no vendor. The create request has no vendor, the entry DTO has no vendor (it has vendors:[{vendor, nodes}] instead, the joined nodes counted by vendor, with an empty vendor for nodes that reported no cards), the form dictionary has no vendor list, and the kube_clusters.vendor column is dropped. An entry has one DaemonSet and no vendor-specific mounts, and the agent Pods run under the default runtime (the agent no longer has a RuntimeClass). The RuntimeClass setting affects only NVIDIA instance Pods; other vendors' devices come through the agent's own device plugin and ignore it.
  2. The agent uses vendor tools through the host root and the host's mount namespace, not through mounts. The agent runs privileged with hostPID, reads the host's files through /proc/1/root, and runs npu-smi, hy-smi, ppu-smi, and nvidia-smi in the host's mount namespace with nsenter, using the host's own C libraries and driver libraries. nvidia-smi is looked up in this order: the host root, /run/nvidia/driver (the NVIDIA GPU Operator's driver container), and last the copy NVIDIA's runtime puts in the agent container (the manifest always sets NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=compute,utility, which only take effect under NVIDIA's runtime). Device-node checks use the host's live /dev.
  3. Unsupported nodes stand by. Kubernetes supports NVIDIA, Ascend, Hygon, and PPU. On a node with no accelerator, or with another vendor's card (AMD, Kunlunxin, and others), the agent Pod runs but does not enroll, so the node never becomes a platform node. The selector remains optional and only narrows which nodes the platform uses.
  4. A node belongs to only one entry. When two entries of the same cluster select the same node, the later one's enrollment is refused (403 KUBE_ENROLL_REFUSED, reason: node_taken, entry the name of the entry holding the node) until the first entry's agent on that node has been offline for more than 15 minutes, after which the node can move to the other entry. This rule makes it safe to change a node selector so that a node moves from one entry to another, and keeps the agents of two DaemonSets from contending for the same node's cards and state directory.
  5. Entries can be edited and uninstalled. PUT /kube/clusters/{uuid} changes every setting except the namespace, the server address, and the state directory; when the manifest changed and the cluster still runs the old one, the response's reapply and (for an entry made by hand) commands say how to catch up. RuntimeClass, the engine port range, and idle return are not written into the manifest: the server sends them to the online agents when a node connects and after the entry is edited (the kube.settings control message; the engine port range rides in the node's network config) and the agents apply them in place, so when only these three change reapply is false. The manifest carries only what an agent needs to start. Uninstalling gives kubectl commands that do not depend on the connect token (remove_commands), and the server runs them for a cloud entry. After the node selector changes, nodes no longer selected go offline, and after 15 minutes another online agent of the same entry confirms and their records are removed automatically.

Why: a cluster often has nodes of several vendors, and one vendor per entry forced users to create several entries, each with its own selector and namespace; vendor-specific mounts (driver library directories, CLI paths, device nodes) vary with the node's distribution and how the driver was installed, and when mismatched the agent starts but finds no cards, the most common source of failed connections. Having the agent run the host's own tools directly on the host removes this class of configuration and failure.

Trade-offs: the agent depends more heavily on the host (hostPID, reading the host root), but it was already privileged; nvidia-smi takes one more lookup step when the container has none injected.

Consequences: one persisted fact fewer, kube_clusters.vendor (a migration drops the column); the KUBE_CLUSTER_MIXED_VENDORS error and the rule that a cloud entry's node pools must be of one vendor (ADR 0063) are removed together; the Nodes page offers a Kubernetes node only the vendor switches supported on Kubernetes.

Amendment (2026-11-12): AMD and Kunlunxin

The vendors that run on Kubernetes are decided by accelerator.RunsOnKube; AMD and Kunlunxin (whole cards) join them, so the "stand by" of item 3 holds only for cards of other vendors. The method follows the previous amendment: device nodes are checked on the host's live /dev through the host root (lstat, glob), xpu-smi is looked up in the host's PATH directories and /usr/local/xpu/bin, and AMD's ROCm version file is read from the host root. Kunlunxin's injection writes no --runtime runc when the driver is kube (a Pod has no such option; it would become a RuntimeClass runc that does not exist). AMD's device plugin names its devices by the card's PCI address, so the kube runtime also matches a device ID to a card by PCI address, besides by UUID. AMD dGPU slicing is not opened to Kubernetes (its runtime constraint is still Docker, nerdctl, and udocker only) (superseded by the 2026-11-14 amendment: AMD and PPU dGPU slicing work on Kubernetes). When one card's device injection fails, that card is not offered, instead of failing the whole node's device plan.

Amendment (2026-11-14): Live Settings, AMD and PPU dGPU, NVIDIA CDI Soft Isolation, Events for Refused Enrollment

  1. RuntimeClass, the engine port range, and idle return are pushed live by the server; they are no longer manifest or agent-side settings. When a node connects (after the initial state replay) and after an entry is edited, the server sends them to every online agent of the entry: RuntimeClass and idle return are the control message kube.settings (internal/proto/kube_settings.go; always the whole current value, never a delta), and the engine port range rides in the node's network config (NetworkConfig.EnginePorts), where the entry's value wins; an offline agent gets them when it reconnects. The agent applies them in place: NVIDIA instance Pods started from then on use the new RuntimeClass (running Pods are unchanged), instance ports allocated from then on come from the new port range (ports of running instances are unchanged), and borrowed cards are returned after the new idle time (switching to permanent borrows the idle cards at once). No reapply, no agent restart. The --engine-ports flag, the GPUWORKS_KUBE_RUNTIME_CLASS environment variable, and the Pod template annotation of an older manifest still work, but only as the initial value until the first message arrives; after that the server's value wins, and they no longer appear the next time the manifest is rendered. So where items 3, 5, and 6 say the agent decides the RuntimeClass or the node reports its own engine port range, this item governs; with the RuntimeClass empty the agent still uses the cluster's nvidia class if there is one and the node's default runtime otherwise.
  2. AMD and PPU dGPU slicing work on Kubernetes. NVIDIA HAMi-core and Ascend vcann-rt could already slice once a whole card was borrowed (item 7); PPU's ppu-slicing-shim and AMD's rocm-slicing-shim now allow the kube runtime too (accelerator.RuntimeConstraint). On other runtimes the cross-process ledger of these two backends is an instance-specific file under the host's /dev/shm, shared through the container's --ipc=host; on Kubernetes the agent's /dev/shm is the agent Pod's private tmpfs, so the agent creates the ledger file (0666) in the host's /dev/shm ahead of time through the host root (/proc/1/root) and mounts that same host path into the instance Pod by hostPath; the ledger path and environment variable do not change, and cleanup removes the file through the host root as well. PPU needs a per-card behavior probe, and on Kubernetes the probe Pod uses the host network (it needs no network and should not depend on the cluster's network plugin, like the instance Pods); AMD is exempt from the behavior probe. Hygon's native vdev still supports only Docker and udocker, is unavailable on Kubernetes, and those cards are deployed whole; Kunlunxin has no dGPU. This item supersedes "AMD dGPU slicing is not opened to Kubernetes" in the previous amendment of the same date.
  3. When NVIDIA's CDI mode leaves no per-card name, fall back to soft isolation instead of failing the Pod. When NVIDIA's container toolkit runs in CDI mode (the GPU Operator's default), it reads a card index or UUID in NVIDIA_VISIBLE_DEVICES as a device of management.nvidia.com/gpu, whose spec holds only all. NVIDIA's own device plugin writes a per-card spec named by UUID when it runs, and the agent uses that name so the Pod gets only its card (strict isolation). Once the cluster uses HAMi or another device plugin in place of NVIDIA's, no CDI spec names the card by UUID (internal/agent/runtime/kube/nvidia_cdi.go): the agent sets the Pod's NVIDIA_VISIBLE_DEVICES to management.nvidia.com/gpu=all, confines the engine to its card with CUDA_VISIBLE_DEVICES (the host card index), and sets CUDA_DEVICE_ORDER=PCI_BUS_ID so the index matches the host's nvidia-smi. The node reports this with its status (NVIDIAIsolation), and the node details show the tag "GPU devices are soft-isolated" with the reason and copyable commands to restore strict isolation (keep the GPU Operator's own device plugin, or switch the toolkit to legacy mode). Trade-off: instances run on such clusters, at the price that the Pod can see every GPU device of the node and only CUDA is restricted, so software that ignores CUDA_VISIBLE_DEVICES or opens a device node directly can reach other cards; an environment that needs strict isolation follows the commands in the tag. Before, the Pod failed to start with unresolvable CDI devices in this situation.
  4. A refused enrollment is written as a Kubernetes event on the agent Pod. Besides logging, the agent records a Warning event (reason GpuworksEnrollRefused) on its own Pod, stating the refusal reason (node_taken, unknown_token, other_cluster) and what to do, visible in the Events of kubectl describe pod -n <namespace> <agent Pod>; while the same reason persists it is refreshed every 10 minutes, and a changed reason is written at once. This needs create on events in the agent's own namespace in the manifest's Role; an older manifest has to be applied again to get it, and without it the event is simply missing while enrollment retries are unaffected.

Why: with the engine port range, RuntimeClass, and idle return in the manifest, changing one meant reapplying it and rolling the DaemonSet, which restarts the agents, even though these are entry settings the server already knows. Soft isolation lets clusters running HAMi and similar connect, instead of the whole node being unusable under CDI mode.

Consequences: no new persisted facts; one more control message between server and agent, kube.settings (an older agent ignores it, and so does a node that is not a Kubernetes node); one more optional field, NVIDIAIsolation, in the node's status report.

Was this page helpful?