How-to Guides
Connect a Kubernetes Cluster
On this page
Use case: You already have a Kubernetes cluster with GPUs and want to hand some of its GPU nodes to gpuworks for model deployment without installing an agent on each machine.
For step-by-step instructions for managed public cloud clusters (Alibaba Cloud ACK, DigitalOcean DOKS, Nebius mk8s), see Connect Managed Kubernetes from a Public Cloud.
Prerequisites:
- The server is running, and the cluster nodes can reach the server address (entered as “Management node address” when connecting; the agent connects outbound; no inbound node port is needed).
- The cluster uses containerd (kubeadm, k3s, RKE2, microk8s, and GPU Operator environments all do). CRI-O (OpenShift) and Docker nodes are not supported; the agent refuses to start and states the reason.
- The vendor driver is installed on the node. Kubernetes supports NVIDIA, Ascend, Hygon (DCU), PPU, AMD, and Kunlunxin, and one cluster entry can cover nodes of all of these vendors; you do not need one entry per vendor (see the next section). On NVIDIA nodes, the NVIDIA container runtime must be available: either as containerd's default runtime (as on managed Kubernetes such as DOKS, Nebius, and ACK), or through a RuntimeClass named
nvidia(k3s creates it automatically). Other vendors only require the vendor driver on the node; they need neither the vendor container runtime nor the vendor device plugin. - Nodes that mount NFS storage have
mount.nfsinstalled (nfs-utils / nfs-common): the agent uses it in the node's mount namespace. - You can run
kubectl applyin the cluster, and aprivilegednamespace is allowed (the manifest labels the namespace with the Pod Security Admissionprivilegedlabel; policies such as Kyverno or Gatekeeper must be relaxed separately).
How It Works
A cluster joins as one DaemonSet. One agent runs on each selected node and exchanges the cluster's connect token for its own node token; from then on it is exactly like an ordinary node: scheduling, deployment, gateway, logs, and terminals make no distinction. An instance becomes a Pod pinned to that node (restartPolicy: Never, host network, model directories mounted by hostPath at the same path), and kubectl get pods -n gpuworks-system shows every instance. Retries and rebuilds remain gpuworks's responsibility; Kubernetes does not restart failed instances.
Each card has only one allocator at a time. When a device plugin (nvidia.com/gpu and similar) is installed on the node, the cards belong to the kubelet: for each card gpuworks wants to use, it starts a placeholder Pod (gpuworks-hold-*) that requests only one card. After all instances have stopped and the cluster's “Idle return” time has elapsed, the placeholder Pods are deleted and the cards go back to Kubernetes. Cards held by other Pods appear in gpuworks as externally occupied and are not scheduled. When the node has no device plugin, gpuworks uses the node's cards directly. When the cluster uses HAMi, ACK GPU sharing, or another way of splitting a card into several shares, borrowing works differently; see the next section.
Coexisting with Other Workloads in the Cluster
gpuworks does not take over the cluster's GPUs. It borrows only the cards it uses, and the cluster's existing workloads keep running:
- The cluster's GPU components stay as they are. gpuworks works with the device plugin, HAMi, or ACK GPU sharing already installed; none of them has to be turned off, replaced, or uninstalled.
- Each scheduler sees what the other holds. A card gpuworks uses is held by a placeholder Pod in Kubernetes, so kube-scheduler does not hand it to another Pod; a card held by another Pod appears in gpuworks as externally occupied, and gpuworks places no instance on it.
- Borrowing involves only the node the instance runs on. The placeholder Pod is pinned to that node, and cards on other nodes are unaffected. An ordinary device plugin and ACK GPU sharing do not accept a request for a specific card, so gpuworks starts one placeholder Pod for each card that is idle on that node at that moment, identifies the one it wants, and returns the rest at once; a new GPU Pod arriving on that node in the meantime stays Pending briefly. HAMi accepts a specific card, so only the wanted card is borrowed.
- Whole cards are borrowed, and gpuworks' own dGPU slicing still works on them. A borrowed card can be sliced among several instances, as on an ordinary node. What cannot be used is a node the cluster itself has already split (MIG, time-slicing, and so on; see the table below).
- Unused cards go back. After the last instance on a card has stopped and the entry's “Idle return” time has elapsed, the placeholder Pod is deleted and the card returns to Kubernetes.
Accelerator Vendor: Detected Automatically
A cluster entry has no vendor setting. An entry has one DaemonSet and no vendor-specific mounts, and the agent Pods run under the default runtime (no RuntimeClass is set for the agent). Each agent detects the accelerator of its own node:
- Vendors supported on Kubernetes: NVIDIA, Ascend, Hygon (DCU), PPU, AMD, Kunlunxin (whole cards; AMD and PPU dGPU slicing is also available on Kubernetes, and Hygon dGPU slicing is not available on Kubernetes, see below).
- A node with no accelerator, or with another vendor's card (vendors other than those above are not supported on Kubernetes yet), stays in stand-by: the agent Pod runs but does not enroll with the server, so such a node never appears on the platform's Nodes page.
- So one entry can cover a cluster whose nodes come from several vendors, and there is no longer a need for one entry per vendor. The node selector remains optional, for narrowing which nodes the platform uses (for example only part of the GPU nodes).
The agent reaches vendor tools without vendor mounts: it runs privileged with hostPID, reads the host's files through the host root (/proc/1/root), and runs the vendor command-line tools npu-smi, hy-smi, ppu-smi, xpu-smi, and nvidia-smi in the host's mount namespace (nsenter), using the host's own C libraries and driver libraries. xpu-smi (Kunlunxin) is looked up in the usual PATH directories and in /usr/local/xpu/bin. AMD needs no command-line tool: the agent reads the host's amdgpu sysfs and KFD, and the ROCm version from the host's /opt/rocm/.info/version. nvidia-smi is looked up in this order: the host root; /run/nvidia/driver (the NVIDIA GPU Operator's driver container); the copy NVIDIA's runtime puts in the agent container (the manifest always sets NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=compute,utility, which only take effect under NVIDIA's runtime). Device-node checks use the host's live /dev.
The NVIDIA GPU Operator's CDI mode (its default) makes the container toolkit read NVIDIA_VISIBLE_DEVICES as CDI device names: an unqualified name such as a card index or UUID falls to management.nvidia.com/gpu, which holds only all, so a pod selecting a card by index does not start (unresolvable CDI devices management.nvidia.com/gpu=0). When the agent creates an instance pod it reads the node's CDI specs (/etc/cdi, /var/run/cdi) through the host root: when the toolkit is in CDI mode (a management.nvidia.com/gpu spec exists and the toolkit's configuration is not in legacy mode), it writes the selected card as the fully qualified device name the NVIDIA device plugin's spec gives it by UUID (such as k8s.device-plugin.nvidia.com/gpu=GPU-…), and the pod gets that card alone.
When no spec names the card (typically the cluster runs HAMi or another device plugin, NVIDIA's own device plugin is not running, and nobody writes the per-card names), the agent no longer fails the pod. It falls back to soft isolation: the pod's NVIDIA_VISIBLE_DEVICES becomes management.nvidia.com/gpu=all (the only device in that spec, which brings in every /dev/nvidiaN of the node), and CUDA_VISIBLE_DEVICES (the card's index on the host, such as 1 or 1,0) confines the engine to the card it was assigned. The agent also sets CUDA_DEVICE_ORDER=PCI_BUS_ID (unless the workload sets it), so that index matches the host's nvidia-smi numbering. Under soft isolation the pod can see every GPU device of the node and only CUDA is restricted; a program that ignores CUDA_VISIBLE_DEVICES or opens the device nodes directly can reach the other cards. The agent reports this with each status report, and the node detail's overview then shows the "GPU devices are soft-isolated" label with the reason and the commands to run (generated by the server; not shown while the node is offline, and back once it is online and has reported again). To return to strict isolation, choose one:
- The cluster uses the GPU Operator and no other GPU device plugin: keep the Operator's own device plugin enabled, which writes the per-card CDI names. Do not install NVIDIA's device plugin next to HAMi or another device plugin; both would advertise the same GPUs.
- Set the toolkit to legacy mode, which isolates by card index.
GPU Operator, enable its own device plugin (when there is no other GPU device plugin):
kubectl patch clusterpolicy "$(kubectl get clusterpolicy -o jsonpath='{.items[0].metadata.name}')" --type=merge -p '{"spec":{"devicePlugin":{"enabled":true}}}'GPU Operator, set the toolkit to legacy mode. When spec.toolkit.env exists, append an entry and keep the existing ones:
kubectl patch clusterpolicy "$(kubectl get clusterpolicy -o jsonpath='{.items[0].metadata.name}')" --type=json -p '[{"op":"add","path":"/spec/toolkit/env/-","value":{"name":"NVIDIA_CONTAINER_RUNTIME_MODE","value":"legacy"}}]'When the previous command fails because spec.toolkit.env does not exist, use this one (it creates the list; an existing list would be replaced):
kubectl patch clusterpolicy "$(kubectl get clusterpolicy -o jsonpath='{.items[0].metadata.name}')" --type=json -p '[{"op":"add","path":"/spec/toolkit/env","value":[{"name":"NVIDIA_CONTAINER_RUNTIME_MODE","value":"legacy"}]}]'Toolkit installed on the node without the GPU Operator: run on that node (default config file /etc/nvidia-container-runtime/config.toml), then restart the container runtime:
sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=legacy
sudo systemctl restart containerdWhen the agent can tell whether the toolkit belongs to the GPU Operator or is installed on the node, the node detail lists only the commands that apply. The service that restarts the container runtime depends on the distribution (containerd, rke2-agent, k3s, and so on).
One Kubernetes node belongs to only one entry at a time. When two entries of the same cluster select the same node, the agent of the second entry is refused enrollment (KUBE_ENROLL_REFUSED, reason node_taken, naming the entry that holds the node) until the first entry's agent on that node has been offline for more than 15 minutes; after that the node can move to the other entry (for example after changing selectors). Other refusal reasons: unknown_token (the connect token is unknown or revoked; an agent that has enrolled before keeps its saved identity and is unaffected) and other_cluster (the token belongs to another Kubernetes cluster). A saved identity records the entry that issued it, and an agent only uses one its own entry issued: when several entries share the default state directory, a node that moves from one entry to another never connects under the previous entry's identity.
When enrollment is refused, besides logging it the agent records a Warning event on its own Pod (reason GpuworksEnrollRefused) stating the refusal reason and what to do about it; it appears in the Events section of kubectl describe pod -n <namespace> <agent Pod>, so there is no need to dig through the agent log. While the same refusal persists the event is refreshed every 10 minutes, and a changed reason (for example another entry now holding the node) is written at once. Writing events needs create on events in the manifest's Role (in the agent's own namespace only); a cluster connected with an older manifest gets this permission only after kubectl apply again, and without it the event is simply missing while enrollment retries are unaffected.
When one Kubernetes cluster is connected through two entries, give the two entries different namespaces: the manifest's objects are named by namespace, so in the same namespace the later apply overwrites the earlier one and the first entry's nodes quietly move to the second. Entries that may be of the same cluster and use the same namespace are marked “Shared namespace”: a cluster connected on its own next to its name in its row of the list, a cluster connected from a cloud account in its group's header (an entry no node has joined yet may be of any cluster and counts; entries known to be of different clusters are not marked). The namespace cannot change after creation: delete one of the entries and create it again with another namespace.
When the Cluster Uses HAMi, ACK GPU Sharing, or Other Splits
How cards are borrowed is chosen automatically from what runs on the node; the cluster entry needs no configuration:
| On the node | What gpuworks does |
|---|---|
A plain device plugin (nvidia.com/gpu, one unit per card) | As above: the placeholder Pod requests 1 card |
HAMi (the node has a hami.io/node-*-register annotation) | The placeholder Pod asks HAMi's scheduler for one whole card and names it: the card wanted is the card borrowed |
ACK GPU sharing (the node has the aliyun.com/gpu-mem resource, or the ack.node.gpu.schedule label is share, cgpu, or core_mem) | The placeholder Pod requests the memory of one whole card, and the card it got is read from the allocation annotation the scheduler writes on the Pod |
| Other splits: MIG, time-slicing / MPS, Ascend vNPU templates without HAMi's registration | No cards are borrowed: all of the node's cards appear as externally occupied, the reason (MIG, time-slicing, vNPU templates, ...) is stated in the card's occupancy information, and no instance is placed on them; they recover automatically once the cluster returns to whole-card allocation |
Once a whole card is borrowed in any of the three modes, dGPU slicing on it (NVIDIA HAMi-core, Ascend vCANN, PPU ppu-slicing-shim, AMD rocm-slicing-shim) is exactly as on an ordinary node: instance Pods request no HAMi or ACK resource, and gpuworks' own dGPU slices a card among several instances. PPU's and AMD's cross-process ledgers reach other runtimes' containers through --ipc=host, sharing the host's /dev/shm directly; on Kubernetes the agent's own /dev/shm is its own pod's private tmpfs, so the agent pre-creates the ledger file on the host's /dev/shm through the host root (/proc/1/root) and hostPath-mounts that same host path into the instance pod, with the ledger path itself unchanged. dGPU slicing for Hygon is verified only on Docker and udocker for now and is not available on Kubernetes nodes; that card is deployed whole.
HAMi: the placeholder Pod is not bound to the node. It sets schedulerName: hami-scheduler and nodeSelector: kubernetes.io/hostname=<node>, requests nvidia.com/gpu: 1, nvidia.com/gpucores: 100, and nvidia.com/gpumem-percentage: 100, and names the card's UUID in the annotation nvidia.com/use-gpuuuid. HAMi's device plugin only serves Pods that went through HAMi's scheduler and carry its allocation annotation, so borrowing cannot bypass the scheduler. Who owns the card is whatever HAMi wrote in hami.io/vgpu-devices-allocated on the placeholder Pod, checked to be the card asked for, with 100% of the compute and as much memory as HAMi registered for the whole card; otherwise the Pod is returned and the reason reported. When other Pods hold part of a card, they appear in gpuworks as externally occupied (Pod name, memory, compute); when the placeholder Pod stays Pending because of that, the borrow fails and the error carries the scheduler's reason (for example CardInsufficientMemory) and the Pods holding the card.
- HAMi's scheduler is named
hami-schedulerby default. If the cluster renamed it, set an environment variable on the agent DaemonSet:kubectl -n gpuworks-system set env ds/gpuworks-agent GPUWORKS_KUBE_HAMI_SCHEDULER=<scheduler name>. With a wrong name no scheduler handles the placeholder Pod and the borrow times out; the error mentions this variable. - HAMi on Ascend follows the shape in HAMi's documentation (request
huawei.com/<model>: 1without memory, and the annotationhami.io/use-<model>-uuid); it has only been verified against test fixtures, not on hardware. - Installing HAMi on Alibaba Cloud ACK takes two more steps: (1) the kube-scheduler the chart generates uses the default leader lease
kube-system/kube-scheduler, which ACK's managed scheduler holds, so HAMi's scheduler never becomes leader and its extender keeps reportingScheduler extender has not become leader yet; with a single replica setscheduler.leaderElect=false; (2) the Kubernetes 1.36 kube-scheduler lists and watchesresourceclaims,resourceslices, anddeviceclassesinresource.k8s.io, and the chart's ClusterRole lacks these read-only permissions, so they must be added.
ACK GPU sharing: supported only on ACK Pro clusters. The placeholder Pod uses the default scheduler, selects this node with nodeSelector, and requests aliyun.com/gpu-mem: <GiB of one card>. One card's memory is the node's aliyun.com/gpu-mem divided by aliyun.com/gpu-count, cross-checked against the label aliyun.accelerator/nvidia_mem; if the cards differ in size or the count does not match the probe, no cards are borrowed and the reason is stated. This mode has no way to name a card, so borrowing works as with a plain device plugin: one placeholder Pod per free card, the card each got read from the card number in scheduler.framework.gpushare.allocation, the wanted ones kept and the rest returned. Other Pods' copy of this annotation records the cards and GiB they hold, and they are shown as externally occupied.
Extra permission the agent needs: HAMi and ACK GPU sharing record who holds how much in annotations on the holding Pods, so the agent reads the Pods on its own node in every namespace (read-only: get and list on pods in the manifest's ClusterRole). Clusters connected with an older manifest get this permission only after kubectl apply again; without it the agent only sees which Pods the kubelet gave cards to, not how much each holds, and borrowing and ownership checks are unaffected.
The node changes mode midway: if the cluster changes the node from nvidia.com/gpu to ACK GPU sharing (or installs HAMi) while the platform holds cards, the new scheduler cannot see the old placeholder Pods and may give the same card to another Pod. When gpuworks notices, those cards appear in gpuworks as externally occupied with the reason “allocated by A when the platform borrowed this one and by B now; Kubernetes may give the card to another pod”, no more cards are borrowed in the old mode, and no new instance is placed on them; instances already on the cards are not affected, and once no instance runs on a card its old placeholder Pod is deleted. When the node goes back to its earlier mode, the cards recover automatically.
The device nodes an instance needs (Ascend /dev/davinci*, Hygon /dev/kfd and /dev/dri/*, PPU /dev/alixpu*, AMD /dev/kfd and /dev/dri/renderD*, Kunlunxin /dev/xpu* and /dev/xpuctrl, and so on) are handed to the instance by the agent's own device plugin: one extended resource per card, gpuworks.ai/accelerator-<card number>, and gpuworks.ai/accelerator-shared for the control nodes shared by all cards. The instance Pod requests the cards it uses and the shared resource, and the kubelet gives the container only those device nodes, so the instance needs no privileged mode. Which device nodes belong to which card comes from the vendor's own device injection, consistent with docker / nerdctl instances on ordinary nodes. Instances that need devices beyond the plugin (for example InfiniBand /dev/infiniband) run privileged.
The resource a card is borrowed through depends on the vendor: NVIDIA nvidia.com/gpu, Ascend huawei.com/Ascend910 and similar, Hygon hygon.com/dcu, PPU alibabacloud.com/ppu, AMD amd.com/gpu, Kunlunxin kunlunxin.com/xpu (baidu.com/xpu on Baidu CCE). AMD's device plugin names its devices by the card's PCI address (for example 0000:43:00.0), and gpuworks matches one to a card by that address (case-insensitive, ignoring the 0000: domain prefix). A Kunlunxin device ID matches a card only when it contains the card's UUID or ends in the card number (so far verified only with test fixtures, not on a real node with the device plugin). When a card's device injection fails verification (a device node is missing, the card is unhealthy), the agent logs the reason and does not offer that card; the node's other cards stay usable.
The agent uses the node's own DNS (dnsPolicy: Default), like an agent installed directly on the host, and does not depend on cluster DNS; the server address must therefore be one the node can resolve, not an in-cluster Service name. The agent runs nerdctl (pulling and importing images, terminals) and mount (NFS) in the node's mount namespace, so the containerd socket and registry configuration (the certs.d generated from k3s's registries.yaml) are used at the node's own locations and need no configuration. When Docker's bundled containerd also runs on the node, the agent identifies the one the kubelet uses from its own container's shim.
Connect
-
On the “External Platforms” page, click “Add External Platform” at the top right and, under “External platform”, choose “Kubernetes cluster” (a self-managed cluster, and a cluster on a cloud that has no account to connect, both go this way); the “Kubernetes cluster” card under “Quick Connect” (shown to administrators only) opens the same dialog with this already chosen. In the dialog, fill in the name and the “Management node address” (the address the agents in the cluster connect to the GPUWorks management node at; it must be reachable from the cluster's Pods, and is not the Kubernetes API server's address), and optionally the node selector (for example
nvidia.com/gpu.present=true; empty means all nodes). There is no vendor field: the agent on each node detects its accelerator itself, so one entry is enough for a cluster with nodes from several vendors; nodes with no accelerator, or one of an unsupported vendor, stay in stand-by and do not become platform nodes. -
Options under “Advanced”:
- State directory: the agent's directory on the node,
/opt/gpuworksby default. When a gpuworks agent is already installed on the node, use a different directory (for example/opt/gpuworks-kube) so the two do not interfere. - containerd socket: when empty, the agent finds it itself (the default locations of containerd, k3s, RKE2, and microk8s); fill it in if it is installed elsewhere.
- API server address: only for clusters where nothing on the node provides Service addresses (no kube-proxy or replacement, such as KubeEdge or OpenYurt edge nodes). Write it as
https://host:port; the address must be in the API server's certificate. When empty, the agent reaches the API server through the cluster'skubernetesService. - Agent image registry: where the nodes pull the agent image from: Docker Hub (the default), Huawei Cloud SWR or quay.io. Clusters in mainland China usually cannot reach Docker Hub; choose Huawei Cloud SWR. The image's tag is the version of the agent's runtime (such as
r1) and does not follow the server's version; see "Upgrade the agent" below. - Agent image: when filled in, your own image runs instead of one from the registry above. For nodes without internet access, push
gpuworks/gpuworks-agent:r1to a private registry the cluster reaches (or import it on each node withctr -n k8s.io images import) and enter that address; nothing needs to be pushed again until the runtime version changes. For the image archive in offline environments, see Deploy in an Offline Environment. - Image pull secret: when the agent image is in a registry that needs a login (an enterprise Harbor or a cloud provider's private registry), first create a docker-registry Secret in the namespace (
kubectl -n gpuworks-system create secret docker-registry regcred …; create the namespace first if it does not exist), then enter its name here. The agent Pod uses it to pull the image, and the placeholder Pods the agent starts reuse it. - RuntimeClass (takes effect at once when saved, see “Edit a Cluster Entry” below): affects only NVIDIA instance Pods, the RuntimeClass they use to reach the GPUs; other vendors' devices come through the agent's own device plugin and ignore it. When empty (the default), each node decides: it uses the cluster's
nvidiaRuntimeClass if there is one, otherwise the node's default runtime. Fill it in only when the RuntimeClass has a different name. - Engine port range (takes effect at once when saved):
40000-42767by default. Do not overlap the cluster's NodePort range (30000-32767by default): kube-proxy diverts NodePort traffic sent to local addresses. - Idle return (takes effect at once when saved): how long a borrowed card is kept after the last instance stops, 60 seconds by default (checked every 30 seconds, so the actual return happens between 60 and 90 seconds); “Keep borrowed cards” means never return them, handing the whole node pool to gpuworks.
- Priority: the value of the PriorityClass (
gpuworks-inference-<namespace>) of gpuworks hold and instance Pods, 10000 by default, from 0 to 1000000000; larger runs first, and ordinary Pods are 0. It also sets the eviction order when a node runs short of memory or disk: lower priority is evicted first. - Allow preemption: off by default (the PriorityClass says
preemptionPolicy: Never): a hold Pod that does not fit only waits, and the scheduler never evicts your other Pods for it. On, with HAMi or ACK GPU sharing (where hold Pods go through the scheduler) the scheduler may evict lower-priority Pods; with a plain device plugin the hold Pods, like all instance Pods, are bound to the node without the scheduler and the switch makes no difference.
- State directory: the agent's directory on the node,
-
Click “Connect”; the dialog shows the
kubectl applycommand to run in the cluster (the button becomes “Close”). Copy it and run it in the cluster:kubectl apply -f 'https://server.example/kube/manifest.yaml?token=…'The selected nodes appear on the Nodes page automatically, carrying the system labels
gpuworks.ai/kubernetes-cluster(the value is the cluster UUID, shown as the cluster name) andgpuworks.ai/kubernetes-node(the node name in Kubernetes); filter the Nodes page by them, or use them in a deployment's node requirements to keep it on one cluster. Nodes added later by scale-out join automatically as well. The cluster takes one row in the “Connected external platforms” list, with “—” in the “Vendor” column and Kubernetes in the “Platform type” column; the number in the “Nodes joined” column opens the Compute Nodes page listing only these nodes. Expanding the row shows “label value” pairs in two columns: how it was connected (“Connected by”), node selector, namespace, card return, creation time, and nodes joined (the count and the split by vendor, such as “Nodes: 2 · NVIDIA ×2”; once nodes have joined, it opens the Compute Nodes page); while no node has joined, the connect command is below them. A “Shared namespace” badge beside the cluster name means another entry uses the same namespace (see Coexisting with Other Workloads in the Cluster above). In the “Actions” column administrators have “Edit” and “Copy connect command” directly, and the rest in the “More” menu: “View manifest”, then below a rule “Uninstall” (see “Uninstall” below), “Revoke”, and “Delete”; revoking and deleting each ask for confirmation in a dialog. If no node has joined 5 minutes after creation, the “Status” column reads “Not connected for long”, and “Troubleshoot” next to it gives thekubectlcommands to run and common causes (the agent image cannot be pulled, the node selector selects no node, the agent cannot reach the server).
The command contains the cluster's connect token and is shown only to administrators. After the connect token is revoked, new nodes can no longer join, and nodes already joined are unaffected.
The “Nodes joined” item of the expanded row lists the joined nodes by vendor (the vendor with the most nodes first; nodes that reported no cards are counted as a group of their own).
Edit a Cluster Entry
On the “External Platforms” page, click “Edit” in the “Actions” column of the row of a cluster connected on its own, or, for a cluster connected from a cloud account, in the actions on the header of that cluster's group under the expanded account (administrators; the endpoint is PUT /api/v1/kube/clusters/{uuid}). It opens the same cluster form used to connect it. You can change: the name, node selector, agent image and image registry, image pull secret, RuntimeClass, containerd socket, API server address, engine port range, idle return, priority, and allow preemption, validated as on creation. The namespace, management node address, and state directory are fixed at creation; to change them, delete the entry and create it again. An entry connected from a cloud account selects nodes by node pool and cannot change the node selector; change its node pools instead with “Change” under “GPU node pools” in the cluster's group (see Connect Managed Kubernetes from a Public Cloud).
RuntimeClass, engine port range, and idle return are not in the manifest, and saving them takes effect at once on the online agents: no apply, no agent restart, no gateway interruption. The server sends them to every online agent of the entry (offline ones get them when they reconnect), and each agent switches in place: NVIDIA instance Pods started from now on use the new RuntimeClass (Pods already running are unchanged); instance ports are allocated from the new range from now on (the ports of running instances do not change); borrowed cards go back after the new idle return time, and when it becomes “Keep borrowed cards” the agent borrows all its idle cards at once. When only these three change, the save response has reapply false and commands empty. Changing the engine port range is allowed while instances run, even when the ports of running instances fall outside the new range: saving succeeds as usual, the instances keep their ports and keep running, and only ports allocated afterwards come from the new range; the response carries one KUBE_ENGINE_PORTS_OUTSIDE in warnings (range: the new range; instances: each with instance_uuid, deployment, and the ports outside the range), which the page shows after saving. Stopped and failed instances do not count, and instances just asked to stop are not listed. An older manifest (still carrying the --engine-ports argument, the GPUWORKS_KUBE_RUNTIME_CLASS environment variable, or the Pod template annotation) keeps working, and the value the server sends wins; those leftovers disappear the next time the manifest is applied for another reason.
Saving any other setting changes only the record in gpuworks. The manifest is rendered from the new settings at once, but the cluster keeps running the old one until it is applied again:
- An entry made by hand (not revoked): after saving, the dialog gives one
kubectlline that deletes the entry's PriorityClass and then applies the manifest (kubectl delete priorityclass gpuworks-inference-<namespace> --ignore-not-found && kubectl apply -f ...): Kubernetes cannot change a PriorityClass's value orpreemptionPolicyin place, only delete and recreate it, and a plainkubectl applyfails when an earlier priority change was never applied, so the deletion always comes along. It is one line so that no time passes without the PriorityClass between the two steps. - An entry connected from a cloud account: click “Apply to the cluster” and the server applies it; when the PriorityClass changed, the server deletes and recreates it itself (the action of that item in the apply result is
replace). - Applying again makes the DaemonSet roll, and the agents restart with it (an agent restart does not restart instance Pods).
Changing the node selector (for a cloud entry, the node pools): after the manifest is applied again, the DaemonSet stops running on nodes that are no longer selected. Before each of those agents exits, it reads its own DaemonSet once, finds the node no longer selected, and returns the cards it borrowed on that node to Kubernetes (deleting the placeholder Pods) before exiting, so none are left holding a card with no agent to manage it; an agent whose node is still selected (an ordinary restart, rolling update or in-place upgrade) is unaffected. The node goes offline once its agent exits. After 15 minutes offline, the platform asks another online agent of the same entry whether the node still exists in Kubernetes and still runs one of the entry's agent Pods. When two checks in a row (about 5 minutes apart, so the gap of a rolling update between deleting and recreating an agent Pod does not count) say no, and the node has no instances, the node record is removed automatically. When the entry has no other online node there is nobody to ask, and nothing is removed automatically: delete the node by hand on the Nodes page (deleting a Kubernetes node needs no SSH). Newly selected nodes join by themselves.
Narrowing the selection needs confirmation: when changing the node selector (for a cloud entry, the node pools) leaves out nodes that still run instances, the server refuses to save with 409 KUBE_SELECTION_DROPS_INSTANCES, and the page opens a confirmation listing those nodes and the deployments on them. When the manifest is applied again, the agents on those nodes are removed; left alone, the instances would keep running and holding cards, but no longer be managed by the platform, reachable through the gateway, or returned. So click “Stop instances and save” (the request carries confirm_drop_instances: true) and the server first asks the instances on those nodes to stop, then saves: only the instances on the leaving nodes stop (recomputed at the moment of confirmation), and replicas of the same deployment on other nodes are left alone; stopping, cordoning, and saving happen in one transaction after every validation (a duplicate name included) has passed, so a save that fails stops no instance; the instances go through the platform's own removal path (state removing, the agent stops the container and reclaims the cards), the request does not wait for them to finish, and saving is not rolled back because of them; a deployment that runs only on those nodes is left without a replica (the confirmation already lists its name). When the server has established that the nodes leave, it also cordons them, so the controller does not rebuild the replicas it just stopped on nodes that are about to lose their agent; a node that is already cordoned or draining keeps its state; the cordon is recorded as an administrator's, so a node that is selected again later stays cordoned until it is uncordoned on the Nodes page; when the nodes could not be verified they may still be selected, so only their instances are stopped and nothing is cordoned. stopped_instances in the response (each with instance_uuid, deployment, node_uuid) lists the instances that were asked to stop, and the page shows that list after saving. The server does not know the nodes' labels in the cluster, so it asks one online agent of the entry to read those nodes and decide whether the new selection still selects them; when no agent can be asked (the entry has no online node, or its agents are too old), every node that runs instances is listed, and the confirmation says it could not be verified and these nodes may be affected (the parameter verified is false); confirming then stops the instances on the nodes listed. Nothing is asked, and no instance is stopped, when the selection does not change, when it only widens, or when the nodes that leave run no instances. Apart from that, saving still changes only gpuworks's record; the cluster changes when the manifest is applied again.
Uninstall
For an entry gpuworks cannot uninstall for you (a cluster connected on its own, or a cloud cluster whose API server has only a private address), its actions include “Uninstall” (visible to administrators, also for revoked entries): a dialog lists the kubectl commands that remove its agents from the cluster; run them in order:
kubectl -n <namespace> delete daemonset gpuworks-agent --ignore-not-found --wait
kubectl delete clusterrolebinding gpuworks-agent-<namespace> --ignore-not-found
kubectl delete clusterrole gpuworks-agent-<namespace> --ignore-not-found
kubectl delete priorityclass gpuworks-inference-<namespace> --ignore-not-found
kubectl delete namespace <namespace> --ignore-not-foundDeleting the DaemonSet deletes the instance Pods and placeholder Pods with it, and the cards go back to Kubernetes. These commands do not need the connect token, whereas kubectl delete -f <manifest address> fetches the manifest with the token and stops working once the token is revoked. For an entry the server can reach (one connected from a cloud account) the server does it for you (“Remove from the cluster”, which checks first and deletes after you confirm). Once the nodes have disappeared from the Nodes page, the entry can be deleted (it is refused while nodes remain, KUBE_CLUSTER_HAS_NODES): clicking “Delete” while nodes remain shows how many are left in the confirmation, with a way to “Remove from the cluster” or “Uninstall”.
Day-to-Day Operations
| Scenario | What to do |
|---|---|
| Maintain a node | Run kubectl cordon in Kubernetes: at the next status report (within a minute) gpuworks likewise stops placing new replicas on that node, and uncordon restores it in the same way; the node's scheduling status shows as stopped by the platform, with the reason “Kubernetes stopped scheduling onto this node” |
| Upgrade the agent | As for any node: select nodes on the Nodes page and click “Upgrade selected Agents”, or click “Upgrade Agent” in the node details. The agent runs from agent/gpuworks-agent in the node's state directory, and an upgrade replaces that program in place, without changing the DaemonSet or reaching a registry; instance Pods are unaffected, and the upgraded program outlives the Pod. The runtime version (the image tag) rarely changes; when a release raises it, its release notes say so, and applying the manifest once more (kubectl apply or “Apply to the cluster”) is enough. The image tag kubectl shows is the runtime version, not the agent's; the Nodes page shows the agent version |
| A node is deleted or scaled in | After a node has been offline for 15 minutes, gpuworks asks another online node of the same cluster to confirm; when the node no longer exists in Kubernetes and has no instances, it is removed automatically |
| Delete a node from gpuworks | Only the record is deleted and the agent is not uninstalled; if the DaemonSet still selects the node, it rejoins, so adjust the node selector first |
| Change the node selector or node pools | See “Edit a Cluster Entry” above: after the manifest is applied again, nodes no longer selected go offline and are cleaned up after 15 minutes; newly selected nodes join by themselves |
| Uninstall | The entry's “Uninstall” gives the kubectl commands (see “Uninstall” above): the DaemonSet, instance Pods, and placeholder Pods are all deleted, and the cards go back to Kubernetes |
Limitations
- Only containerd is supported: images are pulled and imported by the agent through the node's containerd (the kubelet's
k8s.ionamespace), not through the kubelet, so instance images do not use the Pod'simagePullSecrets; credentials for private registries are configured in gpuworks's image sources (the cluster entry's image pull secret covers only the agent image). - Instances use host network and create no Service / Ingress; they are accessed through the gpuworks gateway.
- Vendor command-line tools (
npu-smi,hy-smi,ppu-smi,xpu-smi,nvidia-smi) run in the node's mount namespace and use the node's own C libraries and driver libraries, so the agent image is independent of the node's distribution; the lookup order is in “Accelerator Vendor: Detected Automatically”. - Only NVIDIA, Ascend, Hygon, PPU, AMD, and Kunlunxin nodes are supported; nodes of other vendors stay in stand-by and do not become platform nodes. Hygon and Kunlunxin are deployed as whole cards only: Hygon dGPU memory slicing is not available on Kubernetes, and Kunlunxin has no dGPU; AMD and PPU dGPU slicing is available on Kubernetes.
- The agent runs privileged with hostNetwork and hostPID (required for probing and for running nerdctl and mount through the node's mount namespace).
Was this page helpful?