Skip to main content

ADR 0050: Source Rules Also Cover Model Sites, Proxies Use Label-Based Configuration, and Credentials Are Unified by Host

On this page

Date: 2026-10-17 · Status: Accepted (supersedes the final item under “Consequences” in ADR 0048, “Source rules for model downloads … are a subsequent step”; the other decisions in ADR 0048 remain in effect)

Basis: Model Pull Sources and Pull Proxies Plan.

Background

Image sources are already unified under pull settings (ADR 0048), but model downloads still rely on URL rewrites in network settings, and proxies use global settings plus per-node overrides:

  • A model has only one download address, with no fallback to the original site when a mirror fails. The same address is retried up to 16 times, so a bad mirror can cause a delay of more than an hour. There is no way to express “these nodes may use only the internal network.”
  • The server uses the startup argument --model-endpoint to fetch model metadata and does not read rewrites in network settings; the browser hardcodes hf-mirror.com. A node may be able to download while server-side registration or deployment creation fails to connect to huggingface.co. Deployment admission reads metadata every time and rejects the request on failure.
  • There is no Hugging Face token support, so gated and private models cannot be downloaded.
  • Node overrides replace global settings in full and must be configured one node at a time, with no label support. Images and models have separate settings and separate semantics.

Decisions

  1. Site rules and image rules share one table; match determines the type. A match beginning with http:// or https:// is a site rule (a site root without a path; alternative addresses are http(s) addresses that may include paths); otherwise it is an image rule. The leaf package internal/server/sourceurl is the sole implementation of syntax and matching. Rules are additive: alternative addresses from all matching rules are added to the candidates in rule order and deduplicated, with the site itself last. If any matching rule disables origin fallback, the site itself is discarded; if nothing remains, the result is WEIGHTS_FETCH_FORBIDDEN. Models sourced from directory URLs download only from their own directory and are unaffected by rules. ModelScope and Hugging Face do not fall back to each other.
  2. Calculate candidates in one place. planning.PullSettingsFor retrieves settings applicable to a node, and planning calculates candidates for model file tasks per node (rule alternatives first, then the site). Desired-state replay by the hub and active pushes by the controller use the same entry point; rule changes trigger reevaluation. The server's own metadata candidates come from planning.ServerFetchCandidates.
  3. The agent downloads from candidates in order (internal/agent/modelfetch). Connection, DNS, and TLS errors, as well as any HTTP error status (including 404 for repositories missing from a mirror, and 401/403), put the address into a 30-minute cooldown (placed last, not removed, shared with image pulls) and trigger the next candidate. A single connection receiving no data for 5 minutes counts as one stall; 2 consecutive stalls on the same address trigger the next candidate. Transient errors are retried at most 6 times at addresses that have subsequent candidates. The final address retains 16 retries, and stalls there only trigger retries. When the version is pinned to a commit, partially downloaded files resume after switching addresses; otherwise, downloads restart from the beginning. Progress shows “Downloading from <host>”; after a switch, it includes the previous host and the reason.
  4. Use an explicit checkbox for the control plane, not labels. The server has no node labels, so rules and proxies each have a control_plane column (shown in the UI as “Use for control plane model metadata retrieval” and “Use for control plane outbound access”; the former applies only to site rules). To read metadata, the control plane queries candidates sequentially (5 seconds each) and reads all content from the first candidate that responds. Unreachable metadata does not block registration: the response includes metadata_notice (MODEL_METADATA_UNREACHABLE, listing attempted hosts, or MODEL_METADATA_FORBIDDEN), and the console directs users to the checkbox. Deployment admission reads each model only once and does not reject a deployment because the control plane cannot reach the repository. --model-endpoint becomes an in-process test fixture parameter: one rule per site, pointing to that endpoint, disabling origin fallback, and with the checkbox selected.
  5. Proxies are ordered entries in pull settings; the first match applies, replacing global settings plus node overrides. Entries contain http_proxy, https_proxy, no_proxy, a username and an encrypted password, a node label selector, notes, and a control_plane checkbox. A node uses the first matching entry; entries are not combined (a proxy is selected as a whole, so combining entries has no meaning). If no entry matches, the agent's own environment variables apply. Control plane outbound access (metadata, logos, directory URL registration, image archive URL previews and probes, and credential login tests) uses the first checked entry. The “Pull proxy” card in node details is read-only; writing dockerd settings on docker nodes follows Decision 8 of ADR 0048.
  6. no_proxy uses Go's NO_PROXY semantics, with a single implementation. pkg/httpproxy implements it directly (without introducing golang.org/x/net), using the same semantics as the Go standard library: *, IP/CIDR, domain names matching both themselves and subdomains, a leading . or *. matching only subdomains, and optional ports. Loopback and localhost never use a proxy; the agent appends loopback and the server address. The same list applies to the agent's own requests, nerdctl / udocker command-line pulls (which previously dropped NO_PROXY), and the dockerd drop-in. Directory URL models therefore also use the node proxy; internal file servers must be included in no_proxy.
  7. One credentials table, keyed by host. The new source_credentials table (migration copies the original registry_credentials table and then deletes it) represents “Credentials,” with one entry per host: image registries (username and password, written to a temporary DOCKER_CONFIG) and model sites (token only, with an empty username). A site token is sent as Authorization: Bearer only over https and only when the host exactly matches the credential's host; mirrors and other domains never receive it. Redirects follow Go's behavior, forwarding it to the same domain and its subdomains (such as the site's CDN), but not to other domains. Login tests originate from the control plane through its proxy: /v2/ for registries, and Hugging Face GET /api/whoami-v2 for tokens. The route is /api/v1/pull/credentials; the old /api/v1/pull/registry-credentials* routes return 410.
  8. Browsers use the server's candidates to fetch metadata. When registering or refreshing a Hugging Face model in the console, the browser reads GET /api/v1/pull/metadata-sources?source=huggingface (the control plane's candidates), instead of hardcoding hf-mirror.com. browser_metadata is set only when the browser actually reads content; otherwise, the server reads it.
  9. Offline imports fill in missing information through path registration, without uploads. Copy the model directory to a node or shared storage and register it as a local path, or register an HF / ModelScope model and then use “Register path” on the node. register-path fills in missing model metadata from the local directory (config, README, and size, without overwriting existing values). For directories downloaded with huggingface-cli download --local-dir, if all .cache/huggingface/download/*.metadata files point to the same commit and this is the model's only copy, the model is pinned to that commit. Fully offline clusters add two “forbid” site rules (no alternatives, origin fallback disabled, checkbox selected), so deployments with no available source fail immediately instead of waiting for public network timeouts.
  10. Migration and carryover. Migration converts global URL rewrites to site rules (all nodes, origin fallback allowed, control plane selected) and rewrites in each node override to site rules scoped to gpuworks.ai/node=<uuid>, placed first. Rules are additive, so nodes whose overrides previously replaced global settings now try their own address, the global address, and then the site itself. Proxies are not converted during migration; they are carried over after server startup, when the key is available: migration runs before secretbox and has no encryption key, so it cannot encrypt proxy passwords. During carryover, each node override becomes a node-scoped entry merged with global values and placed first (preserving the proxy that node uses). The global proxy becomes an entry that applies to all nodes with the control plane checkbox selected. Login information embedded in URLs is split into a username and an encrypted password. The old network settings are deleted after completion. Columns for the endpoint and proxy manually selected per download are deleted.
  11. Remove obsolete settings. Remove the “Network” card on the Settings page (proxy, URL rewrites, and DNS suggestions), “Network overrides” in node details, /settings/network, network:<id>, /nodes/{uuid}/network (returns 410), and all network presets (hf-mirror, xget, and DNS). No presets are provided; examples are documented in the how-to.

Trade-offs

  • Put site rules in the same table, rather than a new table: users think in terms of “where to fetch a source,” and images and models have the same structure. The cost is type dispatch based on match during validation and reads (type-dispatched validation in store), with sourceurl and pullref each handling one syntax.
  • Switch addresses and apply cooldown even for 404: to users, a missing repository on a mirror and a failed mirror both mean “cannot download here.” A 30-minute cooldown costs less than retrying every file. The trade-off is that a genuine 404 (a misspelled repository name) fails only after all candidates have been tried.
  • Resume across addresses only for a pinned commit: a branch name may point to different commits on different mirrors, so resuming could produce mixed content. Unpinned versions therefore restart from the beginning when switching addresses.
  • Do not infer control plane applicability automatically: the server has no labels, and an explicit administrator checkbox is more predictable than guessing. The cost is an extra click on first use; connectivity errors direct users to the checkbox.
  • Proxies use the first match instead of combining entries, while rules are additive: proxies select “which route to take”; rules select “which addresses to try.”
  • New persisted facts: site rules reuse the rules table with an added control_plane column. Proxies and credential types are administrator configuration that cannot be derived, so they all use typed columns. Cooldown state is not stored in the database.

Consequences

  • Add /api/v1/pull/proxies, /api/v1/nodes/{uuid}/pull-proxy, /api/v1/pull/credentials, GET /api/v1/pull/metadata-sources, and POST /api/v1/pull/source-rules/test (probes every address in a draft rule for a node); /api/v1/pull/registry-credentials* and /api/v1/nodes/{uuid}/network return 410.
  • Add internal/server/sourceurl (a leaf package) and internal/agent/modelfetch; pkg/httpproxy supports no_proxy.
  • Weight estimates on the deployment page show the candidate order, or “Cannot download” when downloads are forbidden. The scheduling reason is weights_fetch_forbidden; the download API returns 409 WEIGHTS_FETCH_FORBIDDEN; rule changes trigger deployment reevaluation.

Known Limitations

  • The bench tokenizer is still fetched directly from huggingface.co by the agent using http.DefaultClient (pkg/bench/tokenizer/ref.go), bypassing proxies and rules (see todo.md).
  • ModelScope model logos use arbitrary CDN addresses outside the rules; an unreachable logo is silently omitted (see todo.md).
  • Credentials for ModelScope private repositories have not yet been confirmed; only Hugging Face is confirmed so far.

Was this page helpful?