Developer Documentation
Code Map
On this page
The “Architecture Overview” in CLAUDE.md retains only high-value protocol constraints and a list of constraints that you cannot infer from the code. Package-by-package descriptions of responsibilities live here. To understand what a directory or package does, start with this page; to understand why a design works a certain way, where the single source of truth lives, or what is prohibited where, return to CLAUDE.md.
The following terms recur throughout this page:
- The composition root creates modules, supplies dependencies, and coordinates startup and shutdown.
- Reconciliation compares desired state with actual state and drives actual state toward desired state.
- A unit is a service replica that can be independently scheduled, started, failed, retried, and scaled. A single-node replica has one rank; a multi-node replica includes a leader and followers. In PD disaggregation, P (prefill) and D (decode) are each managed as replicas.
- A leaf package sits at the bottom of the dependency graph and provides foundational capabilities for higher layers to reuse. Each entry specifies which modules that package can depend on.
- A single source of truth means that a category of rules, types, or calculations is defined or implemented in one designated location and shared by other modules.
For complete definitions of domain terms such as nodes, deployments, and PD disaggregation, see CONTEXT.md.
Server Side (internal/server/)
-
app: Composition root (config/store/bus/hub/HTTP wiring, startup and graceful shutdown, accesslog, admin bootstrap). -
api(+api/dto): REST layer, split into dozens of files by domain, with strict separation between DTOs and entities; deployment create/evaluate delegate todeployment, and the handlers only decode and map;authhandles JWT login and authentication. -
hub: Agent session management, desired-state delivery (including source rules, proxies, sync group members, and credentials; credentials are delivered only with state.sync), and server_dial. -
Pull settings (ADR 0048, ADR 0050): Typed tables for rules (image rules and model site rules share a table; the type depends on whether
matchincludeshttp(s)://), proxies, sync groups, and credentials live instore(seemigration_pull_settings.gofor migrations); APIs live inapi/pull*.go.planning.PullCandidatesForis the single entry point for image candidates shared by startup, desired-state replay, pulls from the Images page, and multi-node pre-pulls.planning.PullSettingsForretrieves the rules and proxies applicable to a node. Candidates for model file tasks (PlanModelFileTasks, shared by hub replay and controller delivery) and candidates for the server's own metadata reads (ServerFetchCandidates) are also calculated centrally inplanning.controller/image_acquisition.godecides whether a node missing an image should copy it from another node in the same group, and tracks a 3-hour cooldown for node pairs in memory. -
store: GORM entities + versioned migrations (schema_migrations), SQLite (the glebarez driver is used throughout the repository; do not mix it with modernc). -
controller: One reconciliation file per resource, with a handwritten workqueue (workq.go, without client-go). All deployment forms use the same unit reconciliation (units.go/unit_launch.go, ADR 0029); PD-specific startup and drift checks live inpd*.go. Bus events only record what changed inevent_routes.go, which enqueues by dependency every 250 ms: a node event enqueues the deployments with an instance on that node, a model file event enqueues the deployments using that model, and a node storage state change enqueues only deployments that have not settled; a failed read or an object that no longer exists falls back to enqueueing every deployment. Node status has its own 2-second coalescing with pool backoff (node_status.go). -
Automatic placement of multi-node and PD units (ADR 0057):
planning/unit_choice.gopicks nodes (ChooseReplicaNodes,ChoosePDUnits, built onSuggestUnitGroups' groups), andplanning/unit_trial.gois the unit trial plan shared by the API and the controller;api/deployments_auto_units.go(filling in nodes at create and evaluate) anddeployments_auto_scale.go(scaling by replica count, adding PD units,PUT /placement-mode);controller/unit_auto.gomoves only the ranks that do not fit when a unit is about to start (reusing node replacement's candidates and write), andcontroller/pd_pinned.gowritespinned_gpus_unavailablefor a manually placed PD unit whose pinned cards are held;store/units_auto.gomigrates and writes thedeployments.units_autocolumn. -
netplan: Pure leaf package (depends only onproto), and the single decision point for node networking: role/type vocabulary, resolving which network each node uses for each role from attachments and NIC inventories (RDMA network HCA and bootstrap inference live inrdma.go), management network fallback, the parameter network shared by multi-node groups, storage sync paths between two nodes, service-hosting eligibility, and service forwarding addresses. -
network: Network service, and the only entry point for core workflows to ask “which network and address does this role use?” It provides the sole conversion from store rows to netplan models (convert.go), plus validation and changes for networks, attachments, and management network roles.The network catalog is injected through
Catalog:StoreCatalogincludes all networks in the database, whileManagementOnlydeclares no networks and always uses the management network and agent tunnel.appcreates one instance and shares it with hub, controller, proxy, and api. Its supporting leaf packages arenettest(internal network test orchestration) andservingdial(dialing for service network forwarding). -
parallelism: Leaf package and the sole implementation for deployment parallelism (global TP, PP, and node count) and conversion to the number of cards per node. Planning constructs it from placement; scheduler uses it to determine group size, weight allocation, and attention-head divisibility. -
nodelabel: Leaf package (depends only onaccelerator) for the node label vocabulary: the reservedgpuworks.ai/domain and eight system label keys (generated inplanning.SystemLabels), user-written label validation inValidateWritten(shared by node labels and loading networks by label; rejects the reserved domain), and deployment node requirement validation and normalization inNormalizeRequirements(only the eight declared system keys are recognized; values are deduplicated and sorted). Both store and planning depend on it, without needing to depend on each other. -
pdmode: Leaf package for the PD disaggregation mode catalog (engines, roles, connectors, wire protocols, TP rules, ports, transports), and the single source of truth for PD rules.planning/pd*.gouses it to validateplacement.pdand derive PD units and deployment state (instance-to-unit mapping is derived byplanning.MatchUnits, shared by all deployment forms);deploy/pd.goinjects connectors and environments by role;controller/pd*.goprovides PD startup parameters and drift checks;proxy/pd*.gohandles two-stage routing and fallback. -
healthcheck: Deployment health checks. Every minute, sends a minimal request through the gateway for each enabled deployment (proxy.Proxy.Probe, without accounting); the last 10 results remain in memory only.appwires it up and supplies the deployment list. -
pdverify: PD handoff self-check. Performs an internal handoff through the gateway for each prefill instance and identifies the transport method; results are written to the instance timeline. Wired byapp. -
scheduler/planning/estimate: Scheduling pipeline (filters → GPU planner → scorers) + schedule_explain. For GPU memory estimation,pkg/estimatecontains only model-side hardware facts (per-element dtype cost and multimodal tower runtime caches); server-sideestimateis its facade, converting model config + parameters into a VRAM breakdown (deployment form/details) and scheduler request input, and also includes the dtype hardware matrix. Model config.json parsing lives inpkg/modelconfig. The rules for sleeping, yielding and scheduled replica counts each have one implementation inplanning, shared by the API, the automation and the residency arbiter, and not repeated by clients:sleep_level.go(ResolveSleepLeveldecides whether a sleep lands at L1 or L2,OrderSleepchecks eligibility, resolves the level and writes the sleep order; an operator's sleep, a schedule, the automatic sleep, a yield and a yielding wake all go through it),replica_schedule.go(the range of a scheduled replica count and the reason it is refused,ReplicaBounds/ScheduledReplicasRefusal, andScheduleCapabilitiesOf, which gathers them into the deployment DTO'sschedule_capabilities),replica_scaling.go/replica_scaling_units.go(Scaling: adding and removing multi-node and PD replicas, used by the handlers and by the automation's scheduled replica counts), andyield_eligibility.go(the one judgement of whether a deployment may hold a yield policy, shared by the policy endpoint, create and the DTO'syield_policy_eligible). -
sleep: Leaf package for sleep/wake policy (per-instance sleep residue estimated from model parameter count and card count, the L1 host-memory headroom, eligibility checks, and wake admission); does not read store. The scheduling ledger and API only supply snapshots, persistence, and error mapping. The agent measures the residue when the instance is alone on its cards (internal/agent/engine/sleep_residue.go) and the estimate stands in when it cannot. -
automation: The executor behind scheduling analysis. The evaluator carries out schedules and waiting wakes on a 10-second tick, and the residency arbiter's yields when they fall due (an idle time or a turn ending, a wait-to-finish limit running out) on one timer, reading and acting only through theSource/Actionsinterfaces, with actions taking the same lifecycle lock as the API; wired up byapp(automation_adapter.go). -
yieldpolicy: Leaf package (standard library only) and the single source of a deployment's yield policy (never/idle/timeslice/anytime) and its parameters: ranges, defaults and normalization, the yield ruleYieldsand the yield termsTermsOf; shared by store (the column values), the gateway (how long a request may wait for a wake), and placement and the residency arbiter (who may put whom to sleep); see Yield policies. -
deploy: Engine parameter assembly and the instance state machine (includingoptimizingand sleep/wake; state enums are defined ininternal/proto). -
deployment: The deployment use case layer. It creates and evaluates a deployment (a request type in; a deployment row, a preview or a ProductError out;Kindtells a wrong request from a conflict with the cluster as it stands and from a server failure, and api maps it onto an HTTP status) and holds the deployment read model (StoreRows/ListRowsrow lookups,EffectiveState/EffectiveStates, the one derivation of the state a deployment shows, shared by the list, the monitoring center and batches,ModelPreparationmodel preparation observations). It depends on neithernet/httpnorapi(guarded bymake check-architecture); api handlers only decode, call and map, and the automation, batches and MCP reuse the same implementation. Editing, node deletion, storage removal and the benchmark executor are still in api and move in groups. -
proxy:/v1OpenAI-compatible reverse proxy + API keys; sleeping instances return 503. PD handoff lives inpd_sequential.go/pd_concurrent.go.pd_stage.gorecords the request's prefill stage (instance, wait, queueing and Prefill reported by the prefill instance, fallback reason, andpd.prefillspan) in request records and the in-flight table. -
inferenceapi: Stable inference operation vocabulary and Contract (openai.chat/embeddings/rerank, etc.), shared by the catalog, deployment snapshots, proxy, and Playground. -
aihelper: The AI assistant's agent loop (streaming, parallel tools, step budget), task seeds, and known-failure table. Tools read platform APIs in-process as the user asking the question (APIReader, supplied per request byapi, reusing the same routes and authentication):cluster_overview,search_api(the catalogapidocs_gen.gois generated fromdocs/reference/api.mdand guarded by tests),call_api(jq, item-count and length limits, sensitive-field redaction, time conversion), andsearch_metrics. Unreadable GET routes and their reasons are listed inapiread.go.app/aiservice.goassembles each request.The known-failure rule table
signature.goand seed contextseed.goare pure functions. Context assembly for server-side tasks (instance diagnosis, model adaptation, unschedulable deployments, node health, and open-ended questions) lives inapp/aitasks.goandapp/aicontext.go; tool implementations live inapp/aitools.go; the persistent frontend sidebar lives inui/src/assistant/.The estimator produces numeric values; an OpenAI-compatible LLM runs a read-only tool agent loop over the preassembled context. Without an LLM configured, rule matches and adaptation tables remain available.
The same tools are exposed externally through MCP: the protocol lives in the leaf package
mcp(JSON-RPC, version negotiation, method dispatch; it has no knowledge of callers or tools).api/mcp.goonly handles the enable/disable switch, rate limiting, and running tools as the caller (aihelper.Tool.Run, with the same invocation, error reporting, and truncation as the assistant's agent loop). Authentication uses platform API tokens (api/platform_tokens.go):managementAuthrecognizes sessions or tokens, and routes declare token scopes (sessionOnly,readTokensRead; seeapi/auth.go). Validation results are cached briefly instoreand invalidated immediately on revocation. -
provider/provision: Cloud provider (aliyun/digitalocean/runpod/compshare/autodl/vast/zstack) instance rental and SSH prechecks/installation for onboarding.Alibaba Cloud signs V3 requests and installs through Cloud Assistant (
provider.ConnectionCloudAssistant, a connection method that does not use SSH).provision/connectivity.gois the sole implementation of bidirectional node↔server connectivity checks in prechecks. SSH installation (Linux and Mac) uploads the agent andinstallscript.Script, then runs the script (provision/install_script.go;install_command.gogenerates manual installation commands for both curl and offline packages). The script records startup configuration throughgpuworks-agent configure; provision does not assemble separate systemd/nohup startup commands.The Vast, RunPod, and CompShare drivers implement the optional
provider.ServerDialPortMapperinterface (CompShare also implementsprovider.InstancePortMapper, which applies only to container instances) (provider/server_dial_ports.go; each provider'sserver_dial.goreadsprovision.ListeningPortsfor ports already listening on the instance). It selects an available, platform-mapped TCP port for an agent that the server dials into and records its public address; out-of-band recovery also uses it to follow reassigned ports.The optional
DriverFactoryinproviderbinds application-level dependencies. WithInferenceNodeLifecycleProvider, drivers determine node ownership and supply lifecycle handling.provider/zstack/cloud/agent_install.goorganizes host onboarding preparation and shared SSH installation;host_onboarding.gohandles Cloud host checks and initial snapshots.cloud/resourceclaim.Servicehandles claims, release, and reconciliation for selected GPUs; this subpackage maintains the query, mutation, and reconciliation interfaces required for claims.cloud/inventoryholds the host/GPU/capacity types shared by discovery, onboarding, and claims;cloud.Driveradapts resource interfaces and assembles services.cloud/apiclienthandles remote API communication. Deployments do not yet reserve cumulative CPU/memory resources; scheduled reconciliation tasks are not currently started, and automatic recovery after timeouts or interruptions remains unimplemented.appdoes not reference specific provider implementations. -
recovery: Independent SSH bypass recovery for offline agents (deliberately outside controller because SSH can block for tens of seconds). Before login, it rereads the associated cloud provider instance (endpoints.go): this is where reassigned SSH endpoints and mapped server_dial addresses are followed. Each cloud account's instance list is read only once per round (30-second cache). -
installscript: The single source of truth for the agent installation script/install.sh(leaf package). The server-served, SSH-wizard-uploaded, and offline-package-embedded scripts are identical. -
offlinepkg: Builds and serves the offline package/dl/gpuworks-agent.run(self-extracting sh: agents for three platforms +installscript.Script; built on the first request after server startup and stored in<data_dir>/offline/; contains no token). -
tsdb/logstore: Embedded time-series database + FTS5 log search. -
uxrecord/uxstore: Per-request gateway records (leaf package shared by the gateway, request store, and monitoring API) and their storage: a separateux.db, with a single writer goroutine writing UTC daily partitions for request logs (req_; above the per-second budget, all anomalous requests are retained while normal requests are sampled with weights recorded), distribution grids (grid_), and coverage counts (cnt_; the latter two count all requests). Retention deletes whole daily tables. Request tables includetrace_id(not indexed; lookups use the deployment index within a time window; seelookup.go). -
tracecontext: Leaf package for W3C Trace Context: parsingtraceparent/tracestate, time-ordered trace IDs for new gateway traces, span IDs, and ratio sampling based on the low 56 bits of the trace ID. The gateway establishes a gateway span for each forwarded request inproxy/request_identity.goand propagates it to the engine (ADR 0046). -
rollup/statsdb: Long-term statistics for Usage analytics (ADR 0055).rollupaccumulates data in hourly buckets in memory: every completed gateway request (token usage and service quality), every GPU sample ingested into the metrics store, and a once-per-minute snapshot of instances holding GPUs (GPU-hours). It extracts deltas and returns them to the accumulator if a write fails.statsdbmanages the separatestats.dbin the data directory: hourly and daily tables for service quality, GPU utilization, and GPU-hours, daily aggregation, batched retention cleanup, and backfill progress. Token usage resides in the main database (store/usage_stats.go).app/opsstats.goconnects the three: periodic writes, daily aggregation, cleanup across all stores according to data retention settings (store/data_retention.go), a one-time historical backfill from ux.db during upgrades (opsstats_backfill.go), and the implementation ofapi.Analytics. -
schedactivity: The scheduling activity of Scheduling Analysis ("Scheduling Activity" indocs/reference/api.md). The store hands it the scheduling events and the instances' state, sleep and removal events once they are written (store.SetEventObserver; which reason an event counts under and which cards it concerns are defined once, instore/scheduling_activity.go), and it counts the last 25 hours or so by the minute: the events of each reason, each card's switches, each deployment's yields, yields to it and wakes, and the stretches in which at least one of a deployment's replicas is awake and in which one holds cards. On start it is replayed once from the event log (store.ReplaySchedulingActivity); after each hour, the hour is written whole to stats.db's three hour tables.Serviceimplementsapi.SchedulingActivity: the activity route, the sharing stats and the node timelines' stats read it, not the event log;app/schedactivity.gowires it into the statistics loop. -
templateexport: Writes a running deployment as a flat inference template (Save a Deployment as an Inference Template): the launch arguments are read back from the command line the engine actually builds bydeploy.ExportLaunchArgs, which checks that they rebuild the original command; the GPU selector matches only the tested model; the weights name the commit the instance's copy resolved; fields without a reliable source become comments; operations are tested with the health check's smallest request. It does not import store: reading the deployment lives inapp/catalog_templates.go, wired toPOST /deployments/{id}/catalog-templatethroughapi.CatalogTemplates. -
traceexport: Leaf package that retrospectively constructs OTLP spans from per-request gateway records (uxrecord.Request) and sends them to a Collector in batches using OTLP/HTTP JSON + gzip; includes a bounded queue, retries, status counters, and test spans. Configuration comes exclusively from the platform settingtelemetry_export(store/telemetry_export.go, with encrypted header values), applied byapp/telemetry_export.go, which implementsapi.TelemetryService. -
imagecatalog: Image catalog and distribution (including commandmode and community catalogs). Responsibilities are divided as follows:imagecatalog/manifest: Image-domain schema and the system catalog import rulesImportOfficial.imagecatalog/pullref: Pure leaf package and the sole implementation of pull candidates, shared by store, imagecatalog, and planning.Copiesproduces the image's own equivalent copies (templateimage_sources, catalogregistry_sources, original name);PlanCandidatesplaces alternative addresses generated by the node's applicable source rules before those copies and handles disabling origin fallback.imagecatalog/community: Leaf subpackage for community repository cards and tag browsing.Browserholds an injectable HTTP client and tag lists cached by repository configuration fingerprint; the Docker Hub / quay / OCI tag-list APIs live inregistry.go.- The
imagecatalogpackage itself: Embedded seedcatalog-seed.json, startup syncSyncSeed, manual importImportManifest, and database write adapters (seed.go,manifest_sink.go). Update checks read from release channels using thedistribution.Clientinapi.Deps.Outside; seeapi/updates.gofor the API side. - Boundary with store: Store only provides reference counts through
ImageReferenceCountsand deletion/deprecation of missing system rows throughReconcileMissingOfficialImages(store/image_references.go); it does not depend on the manifest package. Store's upsert also ensures that imports do not overwrite operational switches on system rows. - API entry points: Catalog writes and update checks live in
api/images_catalog_import.go; node distribution lives inapi/images_nodes.go.
-
modelmeta: Model metadata. -
internal/server/sourceurl: Pure leaf package (standard library only), and the sole implementation of site-rule syntax, matching, andPlan(combining rules, deduplication, origin fallback decisions).storedispatches validation by type;planningandmodelmetacall it. -
nodeaddr: Node address validation. -
modelcache: Leaf package for eviction plans fornfs_cachestorage-node caches (pure functions: limits, reserved free space, least-recently-used order, and eviction only of unreferenced caches backed by a shared copy); shared by scheduling admission (planning.CachePlan) and controller eviction. Storage and model-file data structures and location queries (store.ModelLocation, the single source of truth for host paths) live instore/storage*.go,model_files.go, andmodel_location.go; see ADR 0034. -
prepobs: Leaf package for in-memory observation of model preparation (bounded, expiring, not persisted): hub holds agent copy-progress frames (Copies, invalidated by session and reverse-indexed by node); controller holds cache eviction phases (Phases). Controller separately records the actual selected preparation nodes (controller/preparation_view.go).planning.PrepareModelViewreduces these together with persisted model-file state into the deployment's preparation view, read by the API throughDeps.Preparation(deployment.ModelPreparation).controller.RetryCacheCopyorchestrates retries of failed cache copies. -
acquisition: Pure-function resolver for display on deployment pages. It lists only storage that already holds the model (present), storage where nodes can save retrieved models (targets), and the template's weight sources (providers); it makes no selections and performs no I/O. How images and models reach a node is determined by pull settings at node startup, not here (ADR 0048). Per-node estimates (nodes) are calculated byplanningduring evaluate. -
inferencecatalog: ADR 0009 flat catalog (original models + list of executable specs),inferencecatalog/seed(built-in packages), andinferencecatalog/packages(content-addressed package repository; the DB stores only digests/projections). -
chipmatch/hardwarecatalog/placement/partitioncap: Matching spec GPU predicates against reported hardware facts, a static accelerator catalog for hypothetical planning, a spec-embedded Starlark topology resolver (deterministic, no I/O, hard limits), and dGPU capability derivation (single source of truth for status/reason text). -
messages: Language-neutral ProductError (stable code + interpolation parameters + English fallback); human-readable text belongs to the client locale. -
webterm: Web terminal session policy (kill switch, concurrency limit, idle timeout, audit). -
agentbin: Embedded agent binaries for each platform (stored with zstd compression;make cross-agentsprefers zstd CLI-19, falling back toscripts/zstdpackwhen unavailable), streamed with decompression and the original length when served through/dl/. -
webui: Frontend build artifacts embedded withgo:embed(internal/server/webui/dist). Text assets are gzipped once on first request and cached in memory; build artifacts no longer include.gzcopies. -
selfinstall/installcred: Implementation ofgpuworks install/upgrade(copying binaries, writing service definitions, retrieving release artifacts from release channels or release sites, and rollback with backups), startup guards for console upgrades, theserve --supervisesupervisor loop without systemd, and initial bootstrap admin credential records (leaf package shared by install and server; must not depend back on server). -
distribution: Release channel client (ADR 0040): signed envelope and index schemas, concurrent requests to all release sources to use the first index that passes signature verification, and sha256-verified files with resumable downloads across sources; schema, validation, local matching, and fetching of publisher notifications innotices.json(notices.go, ADR 0054). Default sources and trusted public keys are constants within the package. Depends only on the standard library (guarded bycheck-architecture) and is shared by selfinstall and api. -
diagnostics: Remote troubleshooting. Debug collection (debug sessions on the server and chosen nodes, plus raw request records for chosen deployments; one at a time, ending by duration or size) and diagnostics collection (logs in a time range, debug collection files, node diagnoses and a redacted database snapshot, packed for download). Its sources are provided byapp/diagnostics_sources.go;gpuworks support-bundleuses the same collection. -
usagereport: User experience improvement program (ADR 0054; seedocs/reference/usage-report.mdfor fields). Leaf package for daily report structures and trimming, in-process counters (API routes/error codes/panics, logins, and agent connection problems, written by theapi.countUsagemiddleware and hub), sending to SLS WebTracking and Cloudflare collection endpoints, daily send times, and the installation-scaleFleetReportattached to update checks. Database-side daily aggregation lives instore/usage_report.go; scheduling and assembly live inapp/usage_report.go; the notification bell service lives inapp/notices.go. -
kubemanifest: Leaf package. Pure-function rendering of the Kubernetes cluster connect manifest (Namespace, RBAC, Secret, PriorityClass, agent DaemonSet). The cluster connect and enroll endpoints are inapi/kube.go, and reclaiming nodes that leave the cluster is inapp/kube_nodes.go(ADR 0052). -
kubeapply: Leaf package. Using a kubeconfig (it trusts only the certificate or token carried in the file), server-side applies the objects rendered bykubemanifestto the API server, or deletes them, with dry-run support and per-object results. The kubeconfig is freshly issued for a single operation byprovider.ClusterProvider(ADR 0063).provider/cluster.gois the cluster capability: listing a cloud account's managed clusters and GPU node pools, issuing kubeconfigs, applying, and removing; the Alibaba Cloud implementation is inprovider/aliyun/ack.go. -
bus: Event publishing. No ORM hooks: writes are consolidated into explicitstoremethods, with publish after transaction commit. -
experiencelaunch: The launcher behind the settings page's experience mode switch: starts and stops a separategpuworks experienceprocess (its own data directory and password) on the console port + 10. Thegpuworks experiencecommand line contract (subcommand and flag names) is defined only here.
Agent Side (internal/agent/)
app: Composition root: reconnection lifecycle + doctor self-checks + background tasks (bench, download retries, netprobe, modeldisk). Connection identity comes from<state>/gpuworks-agent.conf(command-line flags and environment variables can override it; see the CLI reference for precedence); the UUID issued by the server is stored only in memory.conn: Outbound connections + yamux streams; reconnection after disconnects.gpu: Multi-vendor detectors. AMD uses amdgpu sysfs/KFD and injects the selected render device after identity checks; a faulty card does not hide healthy peers. No NVML cgo: usesnvidia-smi+/proc+ sysfs; Ascendnpu-smi, PPUppu-smi, Hygonhy-smi, and Apple siliconsysctl+ioreg.gpu.Proberis injectable for hermetic tests.runtime: Drivers for docker / udocker / nerdctl (containerd, with a fixed namespace for all commands;pull_output.goconverts nerdctl's redrawn progress tables into per-layer byte lines and unpacking lines), plus a recording driver for tests (shells out to CLIs, without Go SDKs);runtime.Driveris injectable.- Container identity label of
runtime/nerdctl:gpuworks.image-targetstores the manifest/index digest looked up before launch.Runappends only this one--labeland keeps the original image reference. Observation looks the image up by target through the internalWorkloadInspection.ImageLookupRef, checks the returned target, and reports the config digest as the container's and image's external ImageID, consistent with Docker and the existing image reports; no separate config-ID label is stored. When the lookup fails, ImageID is not guessed. When the label is missing, invalid, or conflicts with the immutableImage, only the image identity is unknown: the container itself is still observed, and it is not relabeled or recreated. The label survives agent restarts and tag changes after the container is created, but cannot close the window of a concurrent tag change between inspect and create; if the pre-launch lookup fails, the container still starts with the original arguments and no label. Do not write this reserved label by hand. kubeboot: hands a Kubernetes DaemonSet agent over from the image toagent/gpuworks-agentin the node's state directory when it starts (seeded from the image the first time, kept afterwards); a freshly seeded copy takes the server's agent once before it registers, and from then on upgrades in place like any node (ADR 0052 item 8). A leaf package.runtime/kube: Runtime for Kubernetes nodes (ADR 0052). An instance is a bare Pod pinned to this node; images are delegated to nerdctl in thek8s.ionamespace, logs go through the apiserver, and the terminal goes through nerdctl exec.cards.gohandles borrowing cards through placeholder Pods and returning them when idle (including recognizing partial cards);devices.gosplits each card's device nodes between the agent's own device plugin's per-card resources and the shared resource;host.goprepares nerdctl in the node's mount namespace and recognizes the kubelet's containerd by its shim;enroll.goexchanges the connect token for a node token;shutdown.goreads its own DaemonSet and this node before the agent exits, and once the DaemonSet no longer selects this node, deletes every placeholder Pod of this node (cardPool.releaseAllHeld), returning the cards to Kubernetes.kube/kubeapi(a minimal REST client;kubefakeis a faithful fake apiserver for tests),kube/grpcwire(minimal gRPC and protobuf codecs on the standard library),kube/podresources(kubelet PodResources client), andkube/deviceplugin(kubelet device plugin v1beta1: registration, ListAndWatch, Allocate, re-registration after a kubelet restart) are leaf packages. On Kubernetes the agent mounts no vendor directories:gpu.RunCLIsOnHostruns every vendor CLI in the node's mount namespace through the host root (nvidia-smi is looked up on the host, then in/run/nvidia/driver, then in the container),accelerator.RunsOnKubedecides which vendors run on Kubernetes (NVIDIA, Ascend, Hygon, PPU, AMD, Kunlunxin), and nodes with none of their cards stand by without enrolling; device-node checks usestatTool,lstatTool, andglobTool, host filesreadTool, and CLIslookCLI(gpu/hostexec.go). Vendors whose device plugin names devices by PCI address (AMD) are matched to cards throughOptions.CardBDFs.runtime/hostproc: The macOShostdriver (ADR 0044). Images are darwin environment packages (OCI images, pulled and unpacked using the image library inruntime/udocker/native); engines run as host process groups.launch.gotranslates container-style specs (mounts and in-package paths) into host paths.SupportedOnis the sole check enforcing macOS-only availability. Environment packages are built bybuild/host-envs.runtime/procgroup: Records PIDs/exit codes, reaps processes, and stops entire groups for workloads managed without a daemon (hostproc engines and runveil's PRoot), so they can still be found after an agent restart.runtime/runveil: Thin replacement for the udocker CLI (leaf package). Following udocker P1 semantics, constructs argv/env for patched PRoot fromUDOCKER_DIR/containers/<name>, generates passwd/group, and portssetup --nvidia. The udocker driver uses it by default to launch/exec/open terminals/delete containers (GPUWORKS_UDOCKER_LAUNCHER=udockerrestores the CLI launcher). Provisioning and invocation of the udocker CLI are centralized inruntime/udocker/cli.go; with runveil, it is provisioned only on demand. Image pull/import/export/create/metadata operations live inruntime/udocker/native.runveilcliimplementsgpuworks-agent runveil ps/images/load/exec/logs, reading and writing the driver's disk state directly without going through the agent.
- Container identity label of
engine/collector/dgpucache/sysinfo: Engine lifecycle (including per-backend dGPU injectorspartition_{nvidia,ascend,ppu}.go, registered and dispatched by backend ID; container liveness checks after entering running inliveness.go), metrics collection, libvgpu shared-region cache parsing (dGPU utilization reporting), and system information (linux build tag + darwin stub).imagepull: Tries the full candidate list delivered by the server in order (image.pull.candidatescapability). During download, no new bytes for 3 minutes means a stall and advances to the next candidate (only byte counts with layer keys count; layers reporting bytes only in coarse increments receive a relaxed timeout based onStallMinRate; unpacking is not checked for stalls; a download that stays underCrawlMinRatefor a fullCrawlWindowwhile another candidate remains also advances to the next one, but the last candidate and coarse-step layers are not judged this way). When a driver cannot understand runtime progress output, it emitsruntime.PullLineUnreadable, disabling further stall detection for that pull. Driver-generated progress lines must useruntime.PullLine*constructors (pull_lines.go; contract tests with the parser are inimagepull/pull_lines_contract_test.go). Each candidate has a maximum of 60 minutes including unpacking; after a successful pull, the image is tagged locally with the original name. Registry credentials are written to a temporaryDOCKER_CONFIG, which is deleted after the pull.connbudget: Leaf package (standard library only), the connection budget (ADR 0056).Hostsgives each upstream address (registry host, proxy) aPool; one download is aSession, and each request first takes aLease. A session adapts its connection count to its own throughput (doubling from the start, stepping back when the gain is too small), halves on 429/503, refused connections and repeated resets, and honorsRetry-After(an upstream-wide pause that every session waits out).Lease.Observejudges a "crawling connection" by the median of its sibling connections in the same session and by its own peak, replacing at most half per window. Currently only image layer downloads inruntime/udocker/nativeuse it (budget.go); model downloads can reuse it.modelfetch: Downloads model files from server-delivered candidates in order (ADR 0050). Any error status or connection error places the address on a 30-minute cooldown (shared with images). Five minutes without data is a stall; two consecutive stalls advance to the next candidate. An address with subsequent candidates is retried up to 6 times (16 times for the last candidate). When the revision is pinned to a commit, downloads resume across address changes. Site credentials are sent only over https to the same host.registryprobe: Probes registryhttps://<host>/v2/(5-second timeout, 30-minute cache, using the same proxy as pulls), sorts the image's own copies by reachability and latency, and tracks a 30-minute cooldown for failed/stalled endpoints.dockerproxy: Proxy management for docker nodes; writes the node proxy todocker.service.d/zz-gpuworks-proxy.confand runsdaemon-reload(without restarting), and reports dockerd's actual proxy, permissions, and whether the proxy is effective.probe: Behavioral dGPU probes (same injection builder as real instances, quota container probes).terminal: PTY terminal service (host pty with linux build tag / container exec), using the proto terminal frame protocol.storagemount: Leaf package that makes storage available on the local node and reports its status: NFS mounts (injectablemountcommand, without a shell), mountinfo parsing, mount-option allowlist, time-limited filesystem calls with one in-flight call per storage (hung NFS reportsunresponsiveinstead of accumulating calls), test mounts, and control plane heartbeat files.modelscan: Leaf package and the single source of truth for validating a “completed model directory” (ValidateModelDir: readable, contains nonempty files, has no unfinished downloads, and assumes no layout—the layout belongs to the engine selected by the inference template; completion marker name), plus shared-storage scanning (heuristic discovery based on config.json/Mistral layouts, identity determination, traces of incomplete downloads, skip reasons, pagination).download/modelsync: Resumable, multi-connection, rate-limited HTTP downloads (HF/ModelScope) and authenticated direct transfer between nodes (the server only coordinates endpoints and one-time credentials; it does not relay model bytes).netinfo/perftest: NIC inventory (sysfs, read-only without starting processes), host TCP listeners within the engine port range (netlink sock_diag; the kernel returns only LISTEN sockets for scheduling avoidance), and RDMA bandwidth tests.perftestrunsib_write_bw, preferring the host copy and extracting the copy bundled inassetsif unavailable (locate.go), and returns results with reason codes per rail.hostsim/simulate: The experience environment's simulated nodes.hostsimis a leaf package and the one policy of a node that stands in for another machine: it reports host facts, host metrics and engine port occupancy in place of the real host's, and refuses everything that would reach the real host (terminals, files outside the node's directory, ports other than the node's engines, storage, image and model transfer, upgrade, evaluation load, model downloads). Every RPC and stream kind is classified explicitly inrules.go; a kind proto adds without classifying fails a test and is refused at run time. A real node's*hostsim.Hostis nil and every method passes through unchanged, so mainline paths carry one call each and know nothing of simulation.simulateprovides the simulated cards, runtime (it starts no containers and claims ports from the environment's engine host) and that machine'shostsim.Host; it is enabled only byGPUWORKS_SIMULATED_NODEwhen the parent process is the environment that wrote it.
Shared Libraries (pkg/ and Top-Level internal/)
pkg/:
estimate(model-side hardware facts for GPU memory estimation; see above),modelconfig(HF-layout config.json → normalized structure for KV estimation, without binding to either HF or ModelScope as provider),topoparse(NVIDIA/Ascend topology parsing),enginelog(engine log dialect parsing and calibration, with built-in vllm/sglang dialects; the former vllmlog has been merged into it),safetensors(parses safetensors metadata and calculates model weight size),modelrepo(HF/ModelScope repository listing parsers, shared by server+agent, supportingBaseURLand authenticated requests),httpproxy(proxy selection and GoNO_PROXYsemantics forno_proxy, shared by server+agent),bench(benchmark vocabulary and execution: dataset/loadgen/metrics/report/search;search.Documentis the named type for a Run's search column),catalogasset(validation and unpacking of runtime assets bundled with the catalog),localization(BCP 47 wire text types).
Top-level shared packages in internal/:
proto(control plane protocol),engines(single source of truth for engine names; pure leaf with no dependencies:VLLM/SGLang/Callable.Callablehosts catalog-bundled Serve Adapters and has no log dialect or parameter dictionary; useDialectNames()instead ofNames()when dialects/dictionaries are needed.deployre-exports these asEngineVLLM/EngineSGLang/DefaultEngine. Becausedeploydepends onmodelmetathroughinferenceapiand agent cannot depend on server, both sides can share only this leaf package),logging(size-rotated file logging shared by server+agent,--log-file/--log-max-size/--log-keep),logthrottle(log rate limiting shared by server and agent: each kind of record is written a limited number of times per 30 seconds, the rest counted and summarized in one record),logtext(container-output line splitting shared by server and agent: keeps only what follows the last\rin a line, as a terminal shows it, so a progress bar leaves just its final frame; overlong lines are broken instead of ending the log stream withErrTooLong),debuglog(time- and size-bounded debug sessions: while one is open, records at debug and above are also written to a file of their own, the process's own log level unchanged),accelerator(dGPU backend registry + single source of truth for vendor identities:VendorNVIDIA/AMD/Ascend/PPU/Hygonconstants,NormalizeVendor/VendorKey/FrameworkVendor/InferVendor/KnownVendors/RequiresVendorCatalogImage/IsDomestic, plus accelerator onboard network-port capabilities inCardNetworkOf(rail naming, environment variables required by collective communication libraries, and host files required by containers), and the accelerator health vocabulary (health.go: three levels—healthy/warning/fault, stable finding codes, evidence linesFormatHealthEvidence/ParseHealthEvidence;health_guide.go:HealthGuideForderives evidence collection/recheck/remediation commands from vendor + code + BDF). Shared by server+agent; adding a backend requires explicit changes here),assets(embedded native assets: libvgpu, partition probe, ib_write_bw, etc.),commandlog(redacted logs for external commands),templateprobe(core L1/L2/L3 inference template validation),testenv/integrationtest(in-process integration test harnesses),benchrun(execution of one benchmark Run: warmup, closed-loop/open-loop load generation at each point according to the search document, and encoding point results into thestatus.benchmarkprotocol. Shared by server-side load generation, agent-side load generation, andgpuworks-agent bench; each side only supplies destinations for status, warnings, and per-request details),platformmetrics(vocabulary for the platform's own time series: names, units, labels, and meanings of GPU, dGPU, node, network, KV cache disk, benchmark, and gateway operational metrics. Together with the parallel inference vocabularyinferencemetrics, it forms the metric catalog forGET /api/v1/monitor/metrics. Pure leaf, shared by agent and server.TestEveryMetricLiteralIsRegisteredscans agent, server, and UI source code and requires every metric name it encounters to be registered),agentconf(sole definition of agent startup configuration; leaf package: fields, validation, reads/writes, and default listen addressDefaultListenfor<state>/gpuworks-agent.conf, plus shell functionsShellFuncsfor all scripts to read agent configuration on a host, also recognizing the command-line form used by older installations. Shared by the agent'sconfigure/run/smoke/upgradeand the server's installation, prechecks, uninstallation, and out-of-band recovery; see ADR 0051),launchd(single source of truth for Mac agent launchd jobs: label, plist location, plist encoding/decoding, and the installation/removal sequences used byinstall-service/uninstall; pure leaf, shared by agent installation and server discovery of existing agents),version(build version, commit, build time, and release channel shared by server and agent),experience(thegpuworks experienceprocess: a guarded console, the fleet of simulated nodes, the engine host that answers for every simulated instance, an offline HF repository, seed data and light continuous traffic; the guard judges a request by the router's own route pattern, refuses writes by default, and allows reads by default except the routesaihelpermay not read),parentwatch(ends a child process's work once its parent is gone; shared by the experience environment and its simulated nodes).
Auxiliary Commands
cmd/gpuworks-dist: Generates and verifies the release channel directory tree (assemblesigns indexes,verifyreads back from each source,newerprevents out-of-order releases,pruneselects old versions). Called byscripts/publish_dist.sh/make dist-publishand CIrelease-publish; uploads use rclone, and this tool holds no storage credentials.cmd/gpuworks-image-manifest: Generates the system image catalog manifest from the gpuworks-runner build matrix (the embedded seedimagecatalog/catalog-seed.json, which enters the distribution channel with every release). Called bymake image-manifest/make image-manifest-verify; each version'sregistry_sourcesare derived from pullref's built-in rewrite rules, and the registry list is defined only inpullref.cmd/template-probe: CLI for L1/L2/L3 inference template catalog validation (l1|l2-command|l3|classify|alignment|calibrate), for catalog maintainers and CI.cmd/gpuworks-catalog-model-config: Fetches the actual config.json from repositories declared in specs and fills inmodel_config(the estimation subset) in inference catalog packages, eliminating remote fetches at deployment time. Gated repositories requireHF_TOKEN; inaccessible repositories usefallback-sources.json. Reports only by default; writes to disk only with-write.
Test Assets (test/)
l2/(-tags=multiproc, multiple processes using real server/agent binaries),l3/(-tags=docker, container-level),l4/(-tags=cloudgpu,<provider>, cloud/physical machines;l4/cmd/includescleanup-leaked,prepare-compshare-golden,prepare-aliyun-golden, andascend-nerdctl-verify),fake-engine//fake-model/(fake engine images and tiny models),ocifixture(deterministic synthetic OCI images covering edge cases that Dockerfiles cannot express, such as hard links/symbolic links/whiteouts/opaque directories/large layers),runtime-wrapper-*(runtime wrapper images),e2e/(Playwright Python scripts, using real Chrome for Playground/inference catalog acceptance tests; included only in explicit gpu3/L4 jobs, not the offline suite).
Frontend (ui/)
React + Vite; build artifacts are output to internal/server/webui/dist and embedded by the server with go:embed. Parameter dictionaries are delivered by the server's /engines endpoint and are not hardcoded into the frontend. i18n: ui/src/i18n/ (en/zh locales) + ui/i18n/ (glossary and review status); make i18n-check is a prerequisite for make ui.
Was this page helpful?