Skip to main content

Code Map

On this page

The “Architecture Overview” in CLAUDE.md retains only high-value protocol constraints and a list of constraints that you cannot infer from the code. Package-by-package descriptions of responsibilities live here. To understand what a directory or package does, start with this page; to understand why a design works a certain way, where the single source of truth lives, or what is prohibited where, return to CLAUDE.md.

The following terms recur throughout this page:

  • The composition root creates modules, supplies dependencies, and coordinates startup and shutdown.
  • Reconciliation compares desired state with actual state and drives actual state toward desired state.
  • A unit is a service replica that can be independently scheduled, started, failed, retried, and scaled. A single-node replica has one rank; a multi-node replica includes a leader and followers. In PD disaggregation, P (prefill) and D (decode) are each managed as replicas.
  • A leaf package sits at the bottom of the dependency graph and provides foundational capabilities for higher layers to reuse. Each entry specifies which modules that package can depend on.
  • A single source of truth means that a category of rules, types, or calculations is defined or implemented in one designated location and shared by other modules.

For complete definitions of domain terms such as nodes, deployments, and PD disaggregation, see CONTEXT.md.

Server Side (internal/server/)

  • app: Composition root (config/store/bus/hub/HTTP wiring, startup and graceful shutdown, accesslog, admin bootstrap).

  • api (+api/dto): REST layer, split into dozens of files by domain, with strict separation between DTOs and entities; deployment create/evaluate delegate to deployment, and the handlers only decode and map; auth handles JWT login and authentication.

  • hub: Agent session management, desired-state delivery (including source rules, proxies, sync group members, and credentials; credentials are delivered only with state.sync), and server_dial.

  • Pull settings (ADR 0048, ADR 0050): Typed tables for rules (image rules and model site rules share a table; the type depends on whether match includes http(s)://), proxies, sync groups, and credentials live in store (see migration_pull_settings.go for migrations); APIs live in api/pull*.go.

    planning.PullCandidatesFor is the single entry point for image candidates shared by startup, desired-state replay, pulls from the Images page, and multi-node pre-pulls. planning.PullSettingsFor retrieves the rules and proxies applicable to a node. Candidates for model file tasks (PlanModelFileTasks, shared by hub replay and controller delivery) and candidates for the server's own metadata reads (ServerFetchCandidates) are also calculated centrally in planning. controller/image_acquisition.go decides whether a node missing an image should copy it from another node in the same group, and tracks a 3-hour cooldown for node pairs in memory.

  • store: GORM entities + versioned migrations (schema_migrations), SQLite (the glebarez driver is used throughout the repository; do not mix it with modernc).

  • controller: One reconciliation file per resource, with a handwritten workqueue (workq.go, without client-go). All deployment forms use the same unit reconciliation (units.go / unit_launch.go, ADR 0029); PD-specific startup and drift checks live in pd*.go. Bus events only record what changed in event_routes.go, which enqueues by dependency every 250 ms: a node event enqueues the deployments with an instance on that node, a model file event enqueues the deployments using that model, and a node storage state change enqueues only deployments that have not settled; a failed read or an object that no longer exists falls back to enqueueing every deployment. Node status has its own 2-second coalescing with pool backoff (node_status.go).

  • Automatic placement of multi-node and PD units (ADR 0057): planning/unit_choice.go picks nodes (ChooseReplicaNodes, ChoosePDUnits, built on SuggestUnitGroups' groups), and planning/unit_trial.go is the unit trial plan shared by the API and the controller; api/deployments_auto_units.go (filling in nodes at create and evaluate) and deployments_auto_scale.go (scaling by replica count, adding PD units, PUT /placement-mode); controller/unit_auto.go moves only the ranks that do not fit when a unit is about to start (reusing node replacement's candidates and write), and controller/pd_pinned.go writes pinned_gpus_unavailable for a manually placed PD unit whose pinned cards are held; store/units_auto.go migrates and writes the deployments.units_auto column.

  • netplan: Pure leaf package (depends only on proto), and the single decision point for node networking: role/type vocabulary, resolving which network each node uses for each role from attachments and NIC inventories (RDMA network HCA and bootstrap inference live in rdma.go), management network fallback, the parameter network shared by multi-node groups, storage sync paths between two nodes, service-hosting eligibility, and service forwarding addresses.

  • network: Network service, and the only entry point for core workflows to ask “which network and address does this role use?” It provides the sole conversion from store rows to netplan models (convert.go), plus validation and changes for networks, attachments, and management network roles.

    The network catalog is injected through Catalog: StoreCatalog includes all networks in the database, while ManagementOnly declares no networks and always uses the management network and agent tunnel. app creates one instance and shares it with hub, controller, proxy, and api. Its supporting leaf packages are nettest (internal network test orchestration) and servingdial (dialing for service network forwarding).

  • parallelism: Leaf package and the sole implementation for deployment parallelism (global TP, PP, and node count) and conversion to the number of cards per node. Planning constructs it from placement; scheduler uses it to determine group size, weight allocation, and attention-head divisibility.

  • nodelabel: Leaf package (depends only on accelerator) for the node label vocabulary: the reserved gpuworks.ai/ domain and eight system label keys (generated in planning.SystemLabels), user-written label validation in ValidateWritten (shared by node labels and loading networks by label; rejects the reserved domain), and deployment node requirement validation and normalization in NormalizeRequirements (only the eight declared system keys are recognized; values are deduplicated and sorted). Both store and planning depend on it, without needing to depend on each other.

  • pdmode: Leaf package for the PD disaggregation mode catalog (engines, roles, connectors, wire protocols, TP rules, ports, transports), and the single source of truth for PD rules. planning/pd*.go uses it to validate placement.pd and derive PD units and deployment state (instance-to-unit mapping is derived by planning.MatchUnits, shared by all deployment forms); deploy/pd.go injects connectors and environments by role; controller/pd*.go provides PD startup parameters and drift checks; proxy/pd*.go handles two-stage routing and fallback.

  • healthcheck: Deployment health checks. Every minute, sends a minimal request through the gateway for each enabled deployment (proxy.Proxy.Probe, without accounting); the last 10 results remain in memory only. app wires it up and supplies the deployment list.

  • pdverify: PD handoff self-check. Performs an internal handoff through the gateway for each prefill instance and identifies the transport method; results are written to the instance timeline. Wired by app.

  • scheduler / planning / estimate: Scheduling pipeline (filters → GPU planner → scorers) + schedule_explain. For GPU memory estimation, pkg/estimate contains only model-side hardware facts (per-element dtype cost and multimodal tower runtime caches); server-side estimate is its facade, converting model config + parameters into a VRAM breakdown (deployment form/details) and scheduler request input, and also includes the dtype hardware matrix. Model config.json parsing lives in pkg/modelconfig. The rules for sleeping, yielding and scheduled replica counts each have one implementation in planning, shared by the API, the automation and the residency arbiter, and not repeated by clients: sleep_level.go (ResolveSleepLevel decides whether a sleep lands at L1 or L2, OrderSleep checks eligibility, resolves the level and writes the sleep order; an operator's sleep, a schedule, the automatic sleep, a yield and a yielding wake all go through it), replica_schedule.go (the range of a scheduled replica count and the reason it is refused, ReplicaBounds / ScheduledReplicasRefusal, and ScheduleCapabilitiesOf, which gathers them into the deployment DTO's schedule_capabilities), replica_scaling.go / replica_scaling_units.go (Scaling: adding and removing multi-node and PD replicas, used by the handlers and by the automation's scheduled replica counts), and yield_eligibility.go (the one judgement of whether a deployment may hold a yield policy, shared by the policy endpoint, create and the DTO's yield_policy_eligible).

  • sleep: Leaf package for sleep/wake policy (per-instance sleep residue estimated from model parameter count and card count, the L1 host-memory headroom, eligibility checks, and wake admission); does not read store. The scheduling ledger and API only supply snapshots, persistence, and error mapping. The agent measures the residue when the instance is alone on its cards (internal/agent/engine/sleep_residue.go) and the estimate stands in when it cannot.

  • automation: The executor behind scheduling analysis. The evaluator carries out schedules and waiting wakes on a 10-second tick, and the residency arbiter's yields when they fall due (an idle time or a turn ending, a wait-to-finish limit running out) on one timer, reading and acting only through the Source / Actions interfaces, with actions taking the same lifecycle lock as the API; wired up by app (automation_adapter.go).

  • yieldpolicy: Leaf package (standard library only) and the single source of a deployment's yield policy (never / idle / timeslice / anytime) and its parameters: ranges, defaults and normalization, the yield rule Yields and the yield terms TermsOf; shared by store (the column values), the gateway (how long a request may wait for a wake), and placement and the residency arbiter (who may put whom to sleep); see Yield policies.

  • deploy: Engine parameter assembly and the instance state machine (including optimizing and sleep/wake; state enums are defined in internal/proto).

  • deployment: The deployment use case layer. It creates and evaluates a deployment (a request type in; a deployment row, a preview or a ProductError out; Kind tells a wrong request from a conflict with the cluster as it stands and from a server failure, and api maps it onto an HTTP status) and holds the deployment read model (StoreRows/ListRows row lookups, EffectiveState/EffectiveStates, the one derivation of the state a deployment shows, shared by the list, the monitoring center and batches, ModelPreparation model preparation observations). It depends on neither net/http nor api (guarded by make check-architecture); api handlers only decode, call and map, and the automation, batches and MCP reuse the same implementation. Editing, node deletion, storage removal and the benchmark executor are still in api and move in groups.

  • proxy: /v1 OpenAI-compatible reverse proxy + API keys; sleeping instances return 503. PD handoff lives in pd_sequential.go / pd_concurrent.go. pd_stage.go records the request's prefill stage (instance, wait, queueing and Prefill reported by the prefill instance, fallback reason, and pd.prefill span) in request records and the in-flight table.

  • inferenceapi: Stable inference operation vocabulary and Contract (openai.chat/embeddings/rerank, etc.), shared by the catalog, deployment snapshots, proxy, and Playground.

  • aihelper: The AI assistant's agent loop (streaming, parallel tools, step budget), task seeds, and known-failure table. Tools read platform APIs in-process as the user asking the question (APIReader, supplied per request by api, reusing the same routes and authentication): cluster_overview, search_api (the catalog apidocs_gen.go is generated from docs/reference/api.md and guarded by tests), call_api (jq, item-count and length limits, sensitive-field redaction, time conversion), and search_metrics. Unreadable GET routes and their reasons are listed in apiread.go. app/aiservice.go assembles each request.

    The known-failure rule table signature.go and seed context seed.go are pure functions. Context assembly for server-side tasks (instance diagnosis, model adaptation, unschedulable deployments, node health, and open-ended questions) lives in app/aitasks.go and app/aicontext.go; tool implementations live in app/aitools.go; the persistent frontend sidebar lives in ui/src/assistant/.

    The estimator produces numeric values; an OpenAI-compatible LLM runs a read-only tool agent loop over the preassembled context. Without an LLM configured, rule matches and adaptation tables remain available.

    The same tools are exposed externally through MCP: the protocol lives in the leaf package mcp (JSON-RPC, version negotiation, method dispatch; it has no knowledge of callers or tools). api/mcp.go only handles the enable/disable switch, rate limiting, and running tools as the caller (aihelper.Tool.Run, with the same invocation, error reporting, and truncation as the assistant's agent loop). Authentication uses platform API tokens (api/platform_tokens.go): managementAuth recognizes sessions or tokens, and routes declare token scopes (sessionOnly, readTokensRead; see api/auth.go). Validation results are cached briefly in store and invalidated immediately on revocation.

  • provider / provision: Cloud provider (aliyun/digitalocean/runpod/compshare/autodl/vast/zstack) instance rental and SSH prechecks/installation for onboarding.

    Alibaba Cloud signs V3 requests and installs through Cloud Assistant (provider.ConnectionCloudAssistant, a connection method that does not use SSH). provision/connectivity.go is the sole implementation of bidirectional node↔server connectivity checks in prechecks. SSH installation (Linux and Mac) uploads the agent and installscript.Script, then runs the script (provision/install_script.go; install_command.go generates manual installation commands for both curl and offline packages). The script records startup configuration through gpuworks-agent configure; provision does not assemble separate systemd/nohup startup commands.

    The Vast, RunPod, and CompShare drivers implement the optional provider.ServerDialPortMapper interface (CompShare also implements provider.InstancePortMapper, which applies only to container instances) (provider/server_dial_ports.go; each provider's server_dial.go reads provision.ListeningPorts for ports already listening on the instance). It selects an available, platform-mapped TCP port for an agent that the server dials into and records its public address; out-of-band recovery also uses it to follow reassigned ports.

    The optional DriverFactory in provider binds application-level dependencies. With InferenceNodeLifecycleProvider, drivers determine node ownership and supply lifecycle handling. provider/zstack/cloud/agent_install.go organizes host onboarding preparation and shared SSH installation; host_onboarding.go handles Cloud host checks and initial snapshots. cloud/resourceclaim.Service handles claims, release, and reconciliation for selected GPUs; this subpackage maintains the query, mutation, and reconciliation interfaces required for claims. cloud/inventory holds the host/GPU/capacity types shared by discovery, onboarding, and claims; cloud.Driver adapts resource interfaces and assembles services. cloud/apiclient handles remote API communication. Deployments do not yet reserve cumulative CPU/memory resources; scheduled reconciliation tasks are not currently started, and automatic recovery after timeouts or interruptions remains unimplemented. app does not reference specific provider implementations.

  • recovery: Independent SSH bypass recovery for offline agents (deliberately outside controller because SSH can block for tens of seconds). Before login, it rereads the associated cloud provider instance (endpoints.go): this is where reassigned SSH endpoints and mapped server_dial addresses are followed. Each cloud account's instance list is read only once per round (30-second cache).

  • installscript: The single source of truth for the agent installation script /install.sh (leaf package). The server-served, SSH-wizard-uploaded, and offline-package-embedded scripts are identical.

  • offlinepkg: Builds and serves the offline package /dl/gpuworks-agent.run (self-extracting sh: agents for three platforms + installscript.Script; built on the first request after server startup and stored in <data_dir>/offline/; contains no token).

  • tsdb / logstore: Embedded time-series database + FTS5 log search.

  • uxrecord / uxstore: Per-request gateway records (leaf package shared by the gateway, request store, and monitoring API) and their storage: a separate ux.db, with a single writer goroutine writing UTC daily partitions for request logs (req_; above the per-second budget, all anomalous requests are retained while normal requests are sampled with weights recorded), distribution grids (grid_), and coverage counts (cnt_; the latter two count all requests). Retention deletes whole daily tables. Request tables include trace_id (not indexed; lookups use the deployment index within a time window; see lookup.go).

  • tracecontext: Leaf package for W3C Trace Context: parsing traceparent/tracestate, time-ordered trace IDs for new gateway traces, span IDs, and ratio sampling based on the low 56 bits of the trace ID. The gateway establishes a gateway span for each forwarded request in proxy/request_identity.go and propagates it to the engine (ADR 0046).

  • rollup / statsdb: Long-term statistics for Usage analytics (ADR 0055). rollup accumulates data in hourly buckets in memory: every completed gateway request (token usage and service quality), every GPU sample ingested into the metrics store, and a once-per-minute snapshot of instances holding GPUs (GPU-hours). It extracts deltas and returns them to the accumulator if a write fails. statsdb manages the separate stats.db in the data directory: hourly and daily tables for service quality, GPU utilization, and GPU-hours, daily aggregation, batched retention cleanup, and backfill progress. Token usage resides in the main database (store/usage_stats.go). app/opsstats.go connects the three: periodic writes, daily aggregation, cleanup across all stores according to data retention settings (store/data_retention.go), a one-time historical backfill from ux.db during upgrades (opsstats_backfill.go), and the implementation of api.Analytics.

  • schedactivity: The scheduling activity of Scheduling Analysis ("Scheduling Activity" in docs/reference/api.md). The store hands it the scheduling events and the instances' state, sleep and removal events once they are written (store.SetEventObserver; which reason an event counts under and which cards it concerns are defined once, in store/scheduling_activity.go), and it counts the last 25 hours or so by the minute: the events of each reason, each card's switches, each deployment's yields, yields to it and wakes, and the stretches in which at least one of a deployment's replicas is awake and in which one holds cards. On start it is replayed once from the event log (store.ReplaySchedulingActivity); after each hour, the hour is written whole to stats.db's three hour tables. Service implements api.SchedulingActivity: the activity route, the sharing stats and the node timelines' stats read it, not the event log; app/schedactivity.go wires it into the statistics loop.

  • templateexport: Writes a running deployment as a flat inference template (Save a Deployment as an Inference Template): the launch arguments are read back from the command line the engine actually builds by deploy.ExportLaunchArgs, which checks that they rebuild the original command; the GPU selector matches only the tested model; the weights name the commit the instance's copy resolved; fields without a reliable source become comments; operations are tested with the health check's smallest request. It does not import store: reading the deployment lives in app/catalog_templates.go, wired to POST /deployments/{id}/catalog-template through api.CatalogTemplates.

  • traceexport: Leaf package that retrospectively constructs OTLP spans from per-request gateway records (uxrecord.Request) and sends them to a Collector in batches using OTLP/HTTP JSON + gzip; includes a bounded queue, retries, status counters, and test spans. Configuration comes exclusively from the platform setting telemetry_export (store/telemetry_export.go, with encrypted header values), applied by app/telemetry_export.go, which implements api.TelemetryService.

  • imagecatalog: Image catalog and distribution (including commandmode and community catalogs). Responsibilities are divided as follows:

    • imagecatalog/manifest: Image-domain schema and the system catalog import rules ImportOfficial.
    • imagecatalog/pullref: Pure leaf package and the sole implementation of pull candidates, shared by store, imagecatalog, and planning. Copies produces the image's own equivalent copies (template image_sources, catalog registry_sources, original name); PlanCandidates places alternative addresses generated by the node's applicable source rules before those copies and handles disabling origin fallback.
    • imagecatalog/community: Leaf subpackage for community repository cards and tag browsing. Browser holds an injectable HTTP client and tag lists cached by repository configuration fingerprint; the Docker Hub / quay / OCI tag-list APIs live in registry.go.
    • The imagecatalog package itself: Embedded seed catalog-seed.json, startup sync SyncSeed, manual import ImportManifest, and database write adapters (seed.go, manifest_sink.go). Update checks read from release channels using the distribution.Client in api.Deps.Outside; see api/updates.go for the API side.
    • Boundary with store: Store only provides reference counts through ImageReferenceCounts and deletion/deprecation of missing system rows through ReconcileMissingOfficialImages (store/image_references.go); it does not depend on the manifest package. Store's upsert also ensures that imports do not overwrite operational switches on system rows.
    • API entry points: Catalog writes and update checks live in api/images_catalog_import.go; node distribution lives in api/images_nodes.go.
  • modelmeta: Model metadata.

  • internal/server/sourceurl: Pure leaf package (standard library only), and the sole implementation of site-rule syntax, matching, and Plan (combining rules, deduplication, origin fallback decisions). store dispatches validation by type; planning and modelmeta call it.

  • nodeaddr: Node address validation.

  • modelcache: Leaf package for eviction plans for nfs_cache storage-node caches (pure functions: limits, reserved free space, least-recently-used order, and eviction only of unreferenced caches backed by a shared copy); shared by scheduling admission (planning.CachePlan) and controller eviction. Storage and model-file data structures and location queries (store.ModelLocation, the single source of truth for host paths) live in store/storage*.go, model_files.go, and model_location.go; see ADR 0034.

  • prepobs: Leaf package for in-memory observation of model preparation (bounded, expiring, not persisted): hub holds agent copy-progress frames (Copies, invalidated by session and reverse-indexed by node); controller holds cache eviction phases (Phases). Controller separately records the actual selected preparation nodes (controller/preparation_view.go). planning.PrepareModelView reduces these together with persisted model-file state into the deployment's preparation view, read by the API through Deps.Preparation (deployment.ModelPreparation). controller.RetryCacheCopy orchestrates retries of failed cache copies.

  • acquisition: Pure-function resolver for display on deployment pages. It lists only storage that already holds the model (present), storage where nodes can save retrieved models (targets), and the template's weight sources (providers); it makes no selections and performs no I/O. How images and models reach a node is determined by pull settings at node startup, not here (ADR 0048). Per-node estimates (nodes) are calculated by planning during evaluate.

  • inferencecatalog: ADR 0009 flat catalog (original models + list of executable specs), inferencecatalog/seed (built-in packages), and inferencecatalog/packages (content-addressed package repository; the DB stores only digests/projections).

  • chipmatch / hardwarecatalog / placement / partitioncap: Matching spec GPU predicates against reported hardware facts, a static accelerator catalog for hypothetical planning, a spec-embedded Starlark topology resolver (deterministic, no I/O, hard limits), and dGPU capability derivation (single source of truth for status/reason text).

  • messages: Language-neutral ProductError (stable code + interpolation parameters + English fallback); human-readable text belongs to the client locale.

  • webterm: Web terminal session policy (kill switch, concurrency limit, idle timeout, audit).

  • agentbin: Embedded agent binaries for each platform (stored with zstd compression; make cross-agents prefers zstd CLI -19, falling back to scripts/zstdpack when unavailable), streamed with decompression and the original length when served through /dl/.

  • webui: Frontend build artifacts embedded with go:embed (internal/server/webui/dist). Text assets are gzipped once on first request and cached in memory; build artifacts no longer include .gz copies.

  • selfinstall / installcred: Implementation of gpuworks install / upgrade (copying binaries, writing service definitions, retrieving release artifacts from release channels or release sites, and rollback with backups), startup guards for console upgrades, the serve --supervise supervisor loop without systemd, and initial bootstrap admin credential records (leaf package shared by install and server; must not depend back on server).

  • distribution: Release channel client (ADR 0040): signed envelope and index schemas, concurrent requests to all release sources to use the first index that passes signature verification, and sha256-verified files with resumable downloads across sources; schema, validation, local matching, and fetching of publisher notifications in notices.json (notices.go, ADR 0054). Default sources and trusted public keys are constants within the package. Depends only on the standard library (guarded by check-architecture) and is shared by selfinstall and api.

  • diagnostics: Remote troubleshooting. Debug collection (debug sessions on the server and chosen nodes, plus raw request records for chosen deployments; one at a time, ending by duration or size) and diagnostics collection (logs in a time range, debug collection files, node diagnoses and a redacted database snapshot, packed for download). Its sources are provided by app/diagnostics_sources.go; gpuworks support-bundle uses the same collection.

  • usagereport: User experience improvement program (ADR 0054; see docs/reference/usage-report.md for fields). Leaf package for daily report structures and trimming, in-process counters (API routes/error codes/panics, logins, and agent connection problems, written by the api.countUsage middleware and hub), sending to SLS WebTracking and Cloudflare collection endpoints, daily send times, and the installation-scale FleetReport attached to update checks. Database-side daily aggregation lives in store/usage_report.go; scheduling and assembly live in app/usage_report.go; the notification bell service lives in app/notices.go.

  • kubemanifest: Leaf package. Pure-function rendering of the Kubernetes cluster connect manifest (Namespace, RBAC, Secret, PriorityClass, agent DaemonSet). The cluster connect and enroll endpoints are in api/kube.go, and reclaiming nodes that leave the cluster is in app/kube_nodes.go (ADR 0052).

  • kubeapply: Leaf package. Using a kubeconfig (it trusts only the certificate or token carried in the file), server-side applies the objects rendered by kubemanifest to the API server, or deletes them, with dry-run support and per-object results. The kubeconfig is freshly issued for a single operation by provider.ClusterProvider (ADR 0063). provider/cluster.go is the cluster capability: listing a cloud account's managed clusters and GPU node pools, issuing kubeconfigs, applying, and removing; the Alibaba Cloud implementation is in provider/aliyun/ack.go.

  • bus: Event publishing. No ORM hooks: writes are consolidated into explicit store methods, with publish after transaction commit.

  • experiencelaunch: The launcher behind the settings page's experience mode switch: starts and stops a separate gpuworks experience process (its own data directory and password) on the console port + 10. The gpuworks experience command line contract (subcommand and flag names) is defined only here.

Agent Side (internal/agent/)

  • app: Composition root: reconnection lifecycle + doctor self-checks + background tasks (bench, download retries, netprobe, modeldisk). Connection identity comes from <state>/gpuworks-agent.conf (command-line flags and environment variables can override it; see the CLI reference for precedence); the UUID issued by the server is stored only in memory.
  • conn: Outbound connections + yamux streams; reconnection after disconnects.
  • gpu: Multi-vendor detectors. AMD uses amdgpu sysfs/KFD and injects the selected render device after identity checks; a faulty card does not hide healthy peers. No NVML cgo: uses nvidia-smi + /proc + sysfs; Ascend npu-smi, PPU ppu-smi, Hygon hy-smi, and Apple silicon sysctl + ioreg. gpu.Prober is injectable for hermetic tests.
  • runtime: Drivers for docker / udocker / nerdctl (containerd, with a fixed namespace for all commands; pull_output.go converts nerdctl's redrawn progress tables into per-layer byte lines and unpacking lines), plus a recording driver for tests (shells out to CLIs, without Go SDKs); runtime.Driver is injectable.
    • Container identity label of runtime/nerdctl: gpuworks.image-target stores the manifest/index digest looked up before launch. Run appends only this one --label and keeps the original image reference. Observation looks the image up by target through the internal WorkloadInspection.ImageLookupRef, checks the returned target, and reports the config digest as the container's and image's external ImageID, consistent with Docker and the existing image reports; no separate config-ID label is stored. When the lookup fails, ImageID is not guessed. When the label is missing, invalid, or conflicts with the immutable Image, only the image identity is unknown: the container itself is still observed, and it is not relabeled or recreated. The label survives agent restarts and tag changes after the container is created, but cannot close the window of a concurrent tag change between inspect and create; if the pre-launch lookup fails, the container still starts with the original arguments and no label. Do not write this reserved label by hand.
    • kubeboot: hands a Kubernetes DaemonSet agent over from the image to agent/gpuworks-agent in the node's state directory when it starts (seeded from the image the first time, kept afterwards); a freshly seeded copy takes the server's agent once before it registers, and from then on upgrades in place like any node (ADR 0052 item 8). A leaf package.
    • runtime/kube: Runtime for Kubernetes nodes (ADR 0052). An instance is a bare Pod pinned to this node; images are delegated to nerdctl in the k8s.io namespace, logs go through the apiserver, and the terminal goes through nerdctl exec. cards.go handles borrowing cards through placeholder Pods and returning them when idle (including recognizing partial cards); devices.go splits each card's device nodes between the agent's own device plugin's per-card resources and the shared resource; host.go prepares nerdctl in the node's mount namespace and recognizes the kubelet's containerd by its shim; enroll.go exchanges the connect token for a node token; shutdown.go reads its own DaemonSet and this node before the agent exits, and once the DaemonSet no longer selects this node, deletes every placeholder Pod of this node (cardPool.releaseAllHeld), returning the cards to Kubernetes. kube/kubeapi (a minimal REST client; kubefake is a faithful fake apiserver for tests), kube/grpcwire (minimal gRPC and protobuf codecs on the standard library), kube/podresources (kubelet PodResources client), and kube/deviceplugin (kubelet device plugin v1beta1: registration, ListAndWatch, Allocate, re-registration after a kubelet restart) are leaf packages. On Kubernetes the agent mounts no vendor directories: gpu.RunCLIsOnHost runs every vendor CLI in the node's mount namespace through the host root (nvidia-smi is looked up on the host, then in /run/nvidia/driver, then in the container), accelerator.RunsOnKube decides which vendors run on Kubernetes (NVIDIA, Ascend, Hygon, PPU, AMD, Kunlunxin), and nodes with none of their cards stand by without enrolling; device-node checks use statTool, lstatTool, and globTool, host files readTool, and CLIs lookCLI (gpu/hostexec.go). Vendors whose device plugin names devices by PCI address (AMD) are matched to cards through Options.CardBDFs.
    • runtime/hostproc: The macOS host driver (ADR 0044). Images are darwin environment packages (OCI images, pulled and unpacked using the image library in runtime/udocker/native); engines run as host process groups. launch.go translates container-style specs (mounts and in-package paths) into host paths. SupportedOn is the sole check enforcing macOS-only availability. Environment packages are built by build/host-envs.
    • runtime/procgroup: Records PIDs/exit codes, reaps processes, and stops entire groups for workloads managed without a daemon (hostproc engines and runveil's PRoot), so they can still be found after an agent restart.
    • runtime/runveil: Thin replacement for the udocker CLI (leaf package). Following udocker P1 semantics, constructs argv/env for patched PRoot from UDOCKER_DIR/containers/<name>, generates passwd/group, and ports setup --nvidia. The udocker driver uses it by default to launch/exec/open terminals/delete containers (GPUWORKS_UDOCKER_LAUNCHER=udocker restores the CLI launcher). Provisioning and invocation of the udocker CLI are centralized in runtime/udocker/cli.go; with runveil, it is provisioned only on demand. Image pull/import/export/create/metadata operations live in runtime/udocker/native. runveilcli implements gpuworks-agent runveil ps/images/load/exec/logs, reading and writing the driver's disk state directly without going through the agent.
  • engine / collector / dgpucache / sysinfo: Engine lifecycle (including per-backend dGPU injectors partition_{nvidia,ascend,ppu}.go, registered and dispatched by backend ID; container liveness checks after entering running in liveness.go), metrics collection, libvgpu shared-region cache parsing (dGPU utilization reporting), and system information (linux build tag + darwin stub).
  • imagepull: Tries the full candidate list delivered by the server in order (image.pull.candidates capability). During download, no new bytes for 3 minutes means a stall and advances to the next candidate (only byte counts with layer keys count; layers reporting bytes only in coarse increments receive a relaxed timeout based on StallMinRate; unpacking is not checked for stalls; a download that stays under CrawlMinRate for a full CrawlWindow while another candidate remains also advances to the next one, but the last candidate and coarse-step layers are not judged this way). When a driver cannot understand runtime progress output, it emits runtime.PullLineUnreadable, disabling further stall detection for that pull. Driver-generated progress lines must use runtime.PullLine* constructors (pull_lines.go; contract tests with the parser are in imagepull/pull_lines_contract_test.go). Each candidate has a maximum of 60 minutes including unpacking; after a successful pull, the image is tagged locally with the original name. Registry credentials are written to a temporary DOCKER_CONFIG, which is deleted after the pull.
  • connbudget: Leaf package (standard library only), the connection budget (ADR 0056). Hosts gives each upstream address (registry host, proxy) a Pool; one download is a Session, and each request first takes a Lease. A session adapts its connection count to its own throughput (doubling from the start, stepping back when the gain is too small), halves on 429/503, refused connections and repeated resets, and honors Retry-After (an upstream-wide pause that every session waits out). Lease.Observe judges a "crawling connection" by the median of its sibling connections in the same session and by its own peak, replacing at most half per window. Currently only image layer downloads in runtime/udocker/native use it (budget.go); model downloads can reuse it.
  • modelfetch: Downloads model files from server-delivered candidates in order (ADR 0050). Any error status or connection error places the address on a 30-minute cooldown (shared with images). Five minutes without data is a stall; two consecutive stalls advance to the next candidate. An address with subsequent candidates is retried up to 6 times (16 times for the last candidate). When the revision is pinned to a commit, downloads resume across address changes. Site credentials are sent only over https to the same host.
  • registryprobe: Probes registry https://<host>/v2/ (5-second timeout, 30-minute cache, using the same proxy as pulls), sorts the image's own copies by reachability and latency, and tracks a 30-minute cooldown for failed/stalled endpoints.
  • dockerproxy: Proxy management for docker nodes; writes the node proxy to docker.service.d/zz-gpuworks-proxy.conf and runs daemon-reload (without restarting), and reports dockerd's actual proxy, permissions, and whether the proxy is effective.
  • probe: Behavioral dGPU probes (same injection builder as real instances, quota container probes).
  • terminal: PTY terminal service (host pty with linux build tag / container exec), using the proto terminal frame protocol.
  • storagemount: Leaf package that makes storage available on the local node and reports its status: NFS mounts (injectable mount command, without a shell), mountinfo parsing, mount-option allowlist, time-limited filesystem calls with one in-flight call per storage (hung NFS reports unresponsive instead of accumulating calls), test mounts, and control plane heartbeat files.
  • modelscan: Leaf package and the single source of truth for validating a “completed model directory” (ValidateModelDir: readable, contains nonempty files, has no unfinished downloads, and assumes no layout—the layout belongs to the engine selected by the inference template; completion marker name), plus shared-storage scanning (heuristic discovery based on config.json/Mistral layouts, identity determination, traces of incomplete downloads, skip reasons, pagination).
  • download / modelsync: Resumable, multi-connection, rate-limited HTTP downloads (HF/ModelScope) and authenticated direct transfer between nodes (the server only coordinates endpoints and one-time credentials; it does not relay model bytes).
  • netinfo / perftest: NIC inventory (sysfs, read-only without starting processes), host TCP listeners within the engine port range (netlink sock_diag; the kernel returns only LISTEN sockets for scheduling avoidance), and RDMA bandwidth tests. perftest runs ib_write_bw, preferring the host copy and extracting the copy bundled in assets if unavailable (locate.go), and returns results with reason codes per rail.
  • hostsim / simulate: The experience environment's simulated nodes. hostsim is a leaf package and the one policy of a node that stands in for another machine: it reports host facts, host metrics and engine port occupancy in place of the real host's, and refuses everything that would reach the real host (terminals, files outside the node's directory, ports other than the node's engines, storage, image and model transfer, upgrade, evaluation load, model downloads). Every RPC and stream kind is classified explicitly in rules.go; a kind proto adds without classifying fails a test and is refused at run time. A real node's *hostsim.Host is nil and every method passes through unchanged, so mainline paths carry one call each and know nothing of simulation. simulate provides the simulated cards, runtime (it starts no containers and claims ports from the environment's engine host) and that machine's hostsim.Host; it is enabled only by GPUWORKS_SIMULATED_NODE when the parent process is the environment that wrote it.

Shared Libraries (pkg/ and Top-Level internal/)

pkg/:

  • estimate (model-side hardware facts for GPU memory estimation; see above),
  • modelconfig (HF-layout config.json → normalized structure for KV estimation, without binding to either HF or ModelScope as provider),
  • topoparse (NVIDIA/Ascend topology parsing),
  • enginelog (engine log dialect parsing and calibration, with built-in vllm/sglang dialects; the former vllmlog has been merged into it),
  • safetensors (parses safetensors metadata and calculates model weight size),
  • modelrepo (HF/ModelScope repository listing parsers, shared by server+agent, supporting BaseURL and authenticated requests),
  • httpproxy (proxy selection and Go NO_PROXY semantics for no_proxy, shared by server+agent),
  • bench (benchmark vocabulary and execution: dataset/loadgen/metrics/report/search; search.Document is the named type for a Run's search column),
  • catalogasset (validation and unpacking of runtime assets bundled with the catalog),
  • localization (BCP 47 wire text types).

Top-level shared packages in internal/:

  • proto (control plane protocol),
  • engines (single source of truth for engine names; pure leaf with no dependencies: VLLM/SGLang/Callable. Callable hosts catalog-bundled Serve Adapters and has no log dialect or parameter dictionary; use DialectNames() instead of Names() when dialects/dictionaries are needed. deploy re-exports these as EngineVLLM/EngineSGLang/DefaultEngine. Because deploy depends on modelmeta through inferenceapi and agent cannot depend on server, both sides can share only this leaf package),
  • logging (size-rotated file logging shared by server+agent, --log-file/--log-max-size/--log-keep),
  • logthrottle (log rate limiting shared by server and agent: each kind of record is written a limited number of times per 30 seconds, the rest counted and summarized in one record),
  • logtext (container-output line splitting shared by server and agent: keeps only what follows the last \r in a line, as a terminal shows it, so a progress bar leaves just its final frame; overlong lines are broken instead of ending the log stream with ErrTooLong),
  • debuglog (time- and size-bounded debug sessions: while one is open, records at debug and above are also written to a file of their own, the process's own log level unchanged),
  • accelerator (dGPU backend registry + single source of truth for vendor identities: VendorNVIDIA/AMD/Ascend/PPU/Hygon constants, NormalizeVendor/VendorKey/FrameworkVendor/InferVendor/KnownVendors/RequiresVendorCatalogImage/IsDomestic, plus accelerator onboard network-port capabilities in CardNetworkOf (rail naming, environment variables required by collective communication libraries, and host files required by containers), and the accelerator health vocabulary (health.go: three levels—healthy/warning/fault, stable finding codes, evidence lines FormatHealthEvidence/ParseHealthEvidence; health_guide.go: HealthGuideFor derives evidence collection/recheck/remediation commands from vendor + code + BDF). Shared by server+agent; adding a backend requires explicit changes here),
  • assets (embedded native assets: libvgpu, partition probe, ib_write_bw, etc.),
  • commandlog (redacted logs for external commands),
  • templateprobe (core L1/L2/L3 inference template validation),
  • testenv / integrationtest (in-process integration test harnesses),
  • benchrun (execution of one benchmark Run: warmup, closed-loop/open-loop load generation at each point according to the search document, and encoding point results into the status.benchmark protocol. Shared by server-side load generation, agent-side load generation, and gpuworks-agent bench; each side only supplies destinations for status, warnings, and per-request details),
  • platformmetrics (vocabulary for the platform's own time series: names, units, labels, and meanings of GPU, dGPU, node, network, KV cache disk, benchmark, and gateway operational metrics. Together with the parallel inference vocabulary inferencemetrics, it forms the metric catalog for GET /api/v1/monitor/metrics. Pure leaf, shared by agent and server. TestEveryMetricLiteralIsRegistered scans agent, server, and UI source code and requires every metric name it encounters to be registered),
  • agentconf (sole definition of agent startup configuration; leaf package: fields, validation, reads/writes, and default listen address DefaultListen for <state>/gpuworks-agent.conf, plus shell functions ShellFuncs for all scripts to read agent configuration on a host, also recognizing the command-line form used by older installations. Shared by the agent's configure/run/smoke/upgrade and the server's installation, prechecks, uninstallation, and out-of-band recovery; see ADR 0051),
  • launchd (single source of truth for Mac agent launchd jobs: label, plist location, plist encoding/decoding, and the installation/removal sequences used by install-service/uninstall; pure leaf, shared by agent installation and server discovery of existing agents),
  • version (build version, commit, build time, and release channel shared by server and agent),
  • experience (the gpuworks experience process: a guarded console, the fleet of simulated nodes, the engine host that answers for every simulated instance, an offline HF repository, seed data and light continuous traffic; the guard judges a request by the router's own route pattern, refuses writes by default, and allows reads by default except the routes aihelper may not read),
  • parentwatch (ends a child process's work once its parent is gone; shared by the experience environment and its simulated nodes).

Auxiliary Commands

  • cmd/gpuworks-dist: Generates and verifies the release channel directory tree (assemble signs indexes, verify reads back from each source, newer prevents out-of-order releases, prune selects old versions). Called by scripts/publish_dist.sh / make dist-publish and CI release-publish; uploads use rclone, and this tool holds no storage credentials.
  • cmd/gpuworks-image-manifest: Generates the system image catalog manifest from the gpuworks-runner build matrix (the embedded seed imagecatalog/catalog-seed.json, which enters the distribution channel with every release). Called by make image-manifest / make image-manifest-verify; each version's registry_sources are derived from pullref's built-in rewrite rules, and the registry list is defined only in pullref.
  • cmd/template-probe: CLI for L1/L2/L3 inference template catalog validation (l1|l2-command|l3|classify|alignment|calibrate), for catalog maintainers and CI.
  • cmd/gpuworks-catalog-model-config: Fetches the actual config.json from repositories declared in specs and fills in model_config (the estimation subset) in inference catalog packages, eliminating remote fetches at deployment time. Gated repositories require HF_TOKEN; inaccessible repositories use fallback-sources.json. Reports only by default; writes to disk only with -write.

Test Assets (test/)

  • l2/ (-tags=multiproc, multiple processes using real server/agent binaries),
  • l3/ (-tags=docker, container-level),
  • l4/ (-tags=cloudgpu,<provider>, cloud/physical machines; l4/cmd/ includes cleanup-leaked, prepare-compshare-golden, prepare-aliyun-golden, and ascend-nerdctl-verify),
  • fake-engine/ / fake-model/ (fake engine images and tiny models),
  • ocifixture (deterministic synthetic OCI images covering edge cases that Dockerfiles cannot express, such as hard links/symbolic links/whiteouts/opaque directories/large layers),
  • runtime-wrapper-* (runtime wrapper images),
  • e2e/ (Playwright Python scripts, using real Chrome for Playground/inference catalog acceptance tests; included only in explicit gpu3/L4 jobs, not the offline suite).

Frontend (ui/)

React + Vite; build artifacts are output to internal/server/webui/dist and embedded by the server with go:embed. Parameter dictionaries are delivered by the server's /engines endpoint and are not hardcoded into the frontend. i18n: ui/src/i18n/ (en/zh locales) + ui/i18n/ (glossary and review status); make i18n-check is a prerequisite for make ui.

Was this page helpful?