Get started
GPUWorks Documentation
Install GPUWorks, deploy your first model, and learn how to run it in production.
On this page
Where to Start
- New to GPUWorksInstall GPUWorks and Deploy Your First Model, then Console Page Tour
- Trying it locally on a MacRun Your First Local Model on a Mac
- Trying it locally on Windows WSL2Run GPUWorks on Windows WSL2 with Docker
- Going to productionInstall and Upgrade the Control Plane → Production Deployment → Upgrades and Version Compatibility
- Upgrading to a new versionRelease Notes → Upgrades and Version Compatibility
- Having a problemTroubleshooting
- Integrating with the APIOpenAI-Compatible Gateway, Complete REST API Routes
- Contributing to developmentDeveloper Documentation Overview
User Documentation
Tutorials
- Install GPUWorks and Deploy Your First ModelUse case: Install GPUWorks for the first time, connect a GPU node, and deploy a chat model.
- Run Two Models on One GPU: dGPU and Sleep/WakeUse case: Run two small chat models simultaneously on one NVIDIA GPU, set a separate dGPU memory limit for each, and put one of them to sleep and wake it.
- Run Your First Local Model on a MacInstall GPUWorks on an Apple silicon Mac, connect the Mac to the console, choose a model from an inference template, and complete your first local chat. This tutorial covers…
- Scale a Model Service from One Replica to TwoUse case: A model already runs on a single GPU. You want to add an independent replica to share requests, then scale back to one replica when demand drops.
- Deploy a PD Service and Observe the Request PathIn this tutorial, you deploy Qwen/Qwen3-14B on two NVIDIA GPU nodes: Prefill (P) processes prompts, and Decode (D) generates tokens one at a time. You then send a request to…
- Run a Performance Benchmark and Understand the ReportOnce your model service can answer questions, run a performance evaluation with the built-in Fixed-Length Random workload. This tutorial sends requests to your chosen servic…
- Locate a Request with a Slow ResponseWhen a model answers successfully but takes a long time to start or finish its response, use request history to see which stage takes the time. In this tutorial, you make a …
How-To Guides
Installation and Onboarding
- Install and Upgrade the Control PlaneThis guide explains how to install, verify, upgrade, and uninstall the GPUWorks control plane.
- Run GPUWorks on Windows WSL2 with DockerThis page covers what differs from an ordinary Linux host when you run the control plane and a local GPU node inside Windows WSL2 (Ubuntu): the inference runtime must be Doc…
- Upgrades and Version CompatibilityThis guide covers console, command-line, and offline upgrades, along with limitations and rollback methods when versions differ.
- Connect Nodes and Upgrade or Uninstall the AgentConnect a GPU host to a running server. After connecting it, you should see the node online under Compute Nodes and verify its accelerators and runtime.
- Rent Cloud GPUsConnect GPU instances from Alibaba Cloud, RunPod, CompShare, AutoDL, or Vast.ai to GPUWorks, then stop or release the cloud resources when you finish using them. Rent an ins…
- Onboard ZStack CloudAdd an existing ZStack Cloud environment to GPUWorks, synchronize its KVM hosts and GPU resources, and install the agent on the hosts you want to manage.
- Connect a Kubernetes ClusterUse case: You already have a Kubernetes cluster with GPUs and want to hand some of its GPU nodes to gpuworks for model deployment without installing an agent on each machine.
- Connect Managed Kubernetes from a Public Cloud (Alibaba Cloud ACK, DigitalOcean, Nebius)Use case: You rent a managed Kubernetes cluster with GPU nodes on a public cloud and hand it to gpuworks for model deployment. This guide gives the complete steps per cloud …
- Give the Console a Public AddressOpen a temporary public endpoint for a running console on a development machine, GPU node, or internal server behind NAT, and close it when you finish. This guide uses cloud…
- Production DeploymentConfigure an installed server as an entry point for your team or production environment, and check HTTPS, long-lived gateway connections, and credential backups. This guide …
- Configure Node NetworksSpecify which networks nodes with multiple interfaces use for storage sync, compute traffic (multi-node inference and PD KV transfer), and serving traffic, then use internal…
Models and Images
- Use Storage: Local Directories, NFS, and NFS with Local CacheThis guide explains how to add model storage, import an existing model library, select storage for a deployment, and manage local caches.
- Download, Cache, and Sync Models Between NodesThis guide explains how to register a model, download it to a selected storage, or adopt files already on a node so that the model is available for deployment.
- Sync Models and Images Between NodesThis guide explains how to copy existing models or inference images and check the sync results.
- Configure Image and Model Pulls: Source Rules, Proxies, Sync Groups, and CredentialsThis guide explains how to configure model and image source addresses, outbound proxies, node copy scope, and access credentials, and how to verify that the configuration ta…
- Import Models and Images from an HTTP File ServerRegister a model directory or a docker save image archive on an HTTP file server with the platform, then acquire it on a node for deployment. Model directories must provide …
- Deploy in an Offline EnvironmentWhen to use this guide: Your cluster cannot access the public internet: it cannot pull from registries such as Docker Hub or download models from Hugging Face / ModelScope.
Deployment and Inference
- Deploy Models, Select GPUs, and Interpret Scheduling ResultsThis guide explains how to deploy a model as a callable inference service, check placement results, and adjust the node scope after deployment.
- Deploy Models on Kunlun P800P800 PCIe / KL3 uses vendor kunlunxin and framework xre. Validated runtimes are Docker and nerdctl on Linux amd64; runveil is not enabled. This combination supports whole-ca…
- Multi-Node InferenceUse cases: The model does not fit on one node, or you want to distribute TP / PP across nodes so that multiple ranks of a deployment run on different machines.
- Connect MinerU to a Remote Vision ModelGPUWorks deploys the MinerU vision model, manages GPU resources and provides an inference gateway. The MinerU client reads PDFs, renders pages, calls the model and produces …
- PD Disaggregation DeploymentUse cases: With long prompts (thousands of tokens) and high concurrency, prefill competes with decode and makes output unstable. Split inference into separate prefill and de…
- Enable and Troubleshoot dGPUThis guide explains how to check a GPU's available dGPU capabilities, create a deployment with a GPU memory limit, and adjust an existing deployment's quota.
- Put Instances to Sleep and Wake ThemThis guide explains how to put an entire deployment, selected replicas, or a PD role to sleep to release GPU memory, then wake them when needed and verify that service resumes.
- Use Scheduling Analysis: Schedules, Yield Policies, and Waiting WakesThis guide explains how to have the platform carry out sleep, wake, and replica-count changes you agreed on in advance, and how to read the scheduling board to see who holds…
- Reschedule Replicas on Node LossThis guide explains how the platform reschedules the replicas of an offline node onto other nodes: how single-node, multi-node, and PD replicas are handled, how to adjust th…
- Run Batches: the Batch API and the ConsoleThis guide explains how to hand a large set of inference requests to the platform to run asynchronously: prepare a JSONL file, upload it, create a batch, follow its progress…
Operations and Benchmarking
- TroubleshootingUse this guide to troubleshoot node connections, GPUs, models and images, instance startup, and inference requests. Find the section that matches the symptom, confirm the ca…
- View Metrics and Search LogsUse cases: View GPU metrics for clusters and deployments, investigate why an instance fails to start or reports errors, and search historical logs by keyword.
- View Usage AnalyticsUse cases: When providing inference services to external callers, check token usage and cache hits by API key over any time range, review each deployment's success rate and …
- Export Request Traces to OpenTelemetryUse case: Send per-request data from the platform gateway to your tracing system (Jaeger, Grafana Tempo, Grafana Cloud, Datadog, and others). Put your upstream gateway, the …
- Declare Engine Metrics for Inference TemplatesWhen to use this guide: You have written an inference template (especially for a custom or third-party engine with engine: callable), but the inference panels in the deploym…
- Save a Deployment as an Inference TemplateWhen to use this: you have a model running in GPUWorks (the service is up and its API answers) and want to turn that configuration into an inference template, to reuse it yo…
- Run Benchmarks and Interpret ReportsUse cases: Quantify throughput, latency, and success rate for managed deployments or external OpenAI-compatible endpoints, and compare performance across concurrency levels,…
- Use the AI AssistantWhen to use this guide: Find the root cause and fix for an instance startup failure; determine which GPUs in your current cluster can run a model and how many it needs; unde…
- Connect Through MCPConnect GPUWorks to an AI client that supports MCP HTTP transport so the client can read node, deployment, and metric information and answer questions about your cluster. Wh…
Lookup Documentation
Reference
- Console Page TourThe web console is a single-page application bundled with the server binary. This page lists all pages by their group in the left menu. The Chinese source uses the Chinese i…
- Complete REST API RoutesThis document lists the HTTP endpoints for managing GPUWorks resources, including nodes, models, deployments, and settings. Management endpoints use the /api/v1 prefix; auth…
- OpenAI-Compatible GatewayThe server's /v1/* endpoints require API key authentication by default; see Authentication below. The model field in the request body specifies the model name exposed for in…
- CLI and Installation ScriptsGPUWorks provides two command-line programs: gpuworks installs, runs, upgrades, and maintains the control plane; gpuworks-agent connects GPU nodes, checks node environments,…
- Configuration and Environment VariablesThe server has a single configuration source: server.yaml in the installation directory (written by gpuworks install; the unit file and nohup command contain only gpuworks s…
- Control Plane Installation Layout(including agent configuration files on nodes)
- Network Monitoring MetricsThis page is a dictionary of node network and NIXL transfer metrics: their names, labels, measurement scope, and locations in the console. For network configuration, see Con…
- Inference Metrics GlossaryThe platform defines inference metrics only once, in internal/inferencemetrics: short names, types, units, and metrics derived from other metrics. An inference template's ap…
- Request Timeline and Stage BreakdownThe gateway records each forwarded /v1 and Playground request (ADR 0042). The request view, request details, end-to-end latency breakdown, distribution cards, and exported O…
- GPU, Detection, Deployment, and dGPU Support MatrixThe dGPU capability status in the table above is derived from the server's partition capability assessment:
- Ascend CANN Combination MatrixUpdated: 2026-11-13. This document distinguishes three things that are easy to confuse:
- Image Catalog and Inference Image FamiliesGPUWorks assembles the complete engine command itself and does not depend on the image's default entrypoint. The Images page manages runtime images by series → version → nod…
- Sources of Deployment Launch ParametersThis document explains where the inference container's final entrypoint, argv, environment variables, mounts, docker arguments, /dev/shm, ports, network, and devices come fr…
- Executable Inference Catalog PackageInference catalog packages distribute model deployment configurations and accompanying lightweight files. A package can contain multiple models, and each model can provide m…
- Inference Catalog model.yaml Syntax ReferenceThis document provides a field-by-field syntax reference for models/<model-id>/model.yaml. A model can have multiple specs (profiles); each spec describes a deployable confi…
- Security Defaults ChecklistThe management API uses persistent sessions: the browser stores an opaque HttpOnly cookie, and the Bearer JWT is bound to the same session. Every authentication check querie…
- User GlossaryThis is a concise glossary of user-facing terms. For complete definitions and counterexamples, refer to CONTEXT.md in the root directory.
Why It Is Designed This Way
- Architecture Overview: Server, Agent, and yamuxThe server provides the control plane, Web UI, and OpenAI-compatible gateway. The agent runs on each GPU node and handles probing, runtimes, container lifecycles, and metric…
- runveil and udocker: Container Runtimes for Nodes Without DockerIn one sentence: runveil is the container runtime GPUWorks uses on nodes without docker, or where docker cannot expose NVIDIA GPUs to containers. It retains udocker's disk l…
- Typical Deployment TopologiesThis page illustrates seven common GPUWorks deployment layouts: which components connect, in which direction, and which ports you need to open. The diagrams show only the ke…
- Choosing Between dGPU, Sleep/Wake, Multi-Node Inference, Replicas, and PDAfter you get your first model running (see Install GPUWorks and Deploy Your First Model), the next questions are usually: should several models share one GPU, what if a mod…
- Scheduling and GPU Memory EstimationDeployment requests first pass feasibility filters: whether the node is online, framework support, image/model availability, dGPU capabilities, and ability to serve traffic.…
- Yield Policy: How Deployments Share a Set of GPUsThis page explains what a yield policy is, the rule by which the platform makes one deployment give up its cards for another, and the trade-offs behind that rule. For the st…
- Request PriorityRequest priority makes the engine queue requests by priority: higher-priority requests are scheduled first, and when GPU memory (KV cache) runs short, lower-priority request…
- Deployment Launch Configuration SemanticsEvery parameter, environment variable, mount, and command on the deployment page ultimately becomes one launch configuration, and instance startup reads only that configurat…
- SGLang Log Injection: Behavior and BoundariesSGLang's default log format contains only a timestamp and rank prefix, without a level. Actual engine errors therefore look the same as ordinary request logs, so the level f…
- dGPU Semantics and BoundariesdGPU (dynamic GPU) injects GPU memory budgets and the required runtime libraries into containers, constraining multiple instances on the same physical GPU to their respectiv…
- Security Model and Trust BoundariesServer-persisted sessions protect management APIs: the web UI uses HttpOnly cookies, and non-browser clients can use session-bound JWTs. Authentication checks session validi…
- Architecture Decision Records (ADRs)This index lists GPUWorks architecture decisions, their status, and which decisions supersede them. Moved-out decisions retain their original numbers.
Release Notes
Developer Documentation
- Developer Documentation OverviewGPUWorks consists of two static Go binaries built with CGO_ENABLED=0 and a React frontend. The control plane entry point is cmd/gpuworks, and the node entry point is cmd/gpu…
- Code Map: Package ResponsibilitiesThe “Architecture Overview” in CLAUDE.md retains only high-value protocol constraints and a list of constraints that you cannot infer from the code. Package-by-package descr…
- Local SetupA local server started with make run-server listens on :8800 by default and writes data to ./data. Parallel development instances use 8800, 8802, 8804, and 8806 in order. Us…
- Testing Guide and Cloud Test TiersCLAUDE.md is the sole authority for testing requirements. This guide explains how to select and run tests at each layer; it does not redefine quality gates.
- AMD Integration and Hardware ValidationSee AMD support design for the implementation design.
- Benchmark Information and Reporting StandardsThis standard governs information retention, statistical definitions, interpretation, and report export for performance benchmarking, covering load generation, server/agent …
- Observability Limits of Per-Request Engine Queue TimeA monitored request's engine_queue_ms must come from timing reported in that same inference response. Queue counts in /metrics are instantaneous gauges, and service histogra…
- Browser E2E Testing StandardsView source on GitHub
- Releases and Embedded AssetsRun the following commands from the repository root. Build prerequisite artifacts such as the UI first, then generate the archives for distribution:
- Contribution GuidePrepare a development branch, verify your changes, and submit a contribution.
- PR / MR WorkflowPR / MR template
Was this page helpful?