How-to Guides
Declare Engine Metrics for an Inference Template
On this page
When to use this guide: You have written an inference template (especially for a custom or third-party engine with engine: callable), but the inference panels in the deployment details page, instance page, and service monitoring page show no data after deployment. Or you want the engine's own TTFT, KV utilization, and speculative acceptance rate to appear in platform panels.
After completing this guide, the template's metric mappings pass validation, and you can see the corresponding data in the platform panels or metrics explorer after an actual inference request.
Prerequisites: The engine provides a Prometheus text or JSON metrics endpoint that the node's agent can reach through the instance port. You can sign in to that node to inspect the raw response, and you can import or edit inference templates (see the Inference Catalog model.yaml Syntax Reference).
Decide Whether a Declaration Is Needed
First use the table below to decide whether you need to modify the template. Every 10 seconds, the agent scrapes the endpoints of running instances and converts the results to platform metrics using mappings. When api.metrics is not declared:
| Template engine | Default behavior | Is a declaration needed? |
|---|---|---|
vllm / sglang | Uses the platform's built-in vLLM / SGLang mappings | No; to add or change a few mappings, specify only map |
callable, with metrics prefixed by vllm: or sglang: (a modified vLLM / SGLang engine or an adapter that wraps SGLang) | Scrapes /metrics and automatically applies the vLLM or SGLang mappings based on the detected prefix | Usually not |
callable, with the engine's own metric names (such as TensorFold's tensorfold:) | Scrapes /metrics, but no names match, so no engine metrics are produced | Yes |
The engine's metrics endpoint returns JSON (such as /metrics in Strata 0.1.38) | The Prometheus text parser cannot parse any lines | Declare format: json; see below |
| The engine has no metrics endpoint | Scrape failures are skipped | A declaration does not help; the gateway collects latency, token, and similar statistics |
When a callable template does not declare api.metrics, importing it (or prechecking it with POST /api/v1/inference-templates/evaluate) returns the warning CATALOG_CALLABLE_METRICS_UNDECLARED. The UI lists this warning after import or save. It does not block import and can be ignored if the engine has no metrics endpoint or uses vLLM / SGLang metric names (see Pre-import Checks and Warnings).
1. Inspect the Engine's Raw Metrics
On the node hosting the instance, request the instance port directly (find the port in the instance details; the agent uses this same address for scraping):
curl --silent --show-error --include http://127.0.0.1:<instance-port>/metricsIf the engine's endpoint is not /metrics, replace it with the actual path first. Confirm that the HTTP status indicates success, then inspect the response headers and body. Do not filter out # lines yet; you need # TYPE later to determine metric types.
For runveil nodes, also run the request on the host. If curl is available inside the container, you can instead run gpuworks-agent runveil exec <container-name-prefix> curl --silent --show-error --include http://127.0.0.1:<instance-port>/metrics (see the CLI Reference).
Check the response's Content-Type: application/json indicates a JSON endpoint; write its mappings as described under JSON Endpoints. For Prometheus text, inspect the output to confirm three things: the family name (tensorfold:requests_running), the type (the # TYPE line; histograms have three sets of lines: _bucket / _sum / _count), and labels (whether a family has multiple lines distinguished by labels such as tier or state).
2. Write the Mappings
Add metrics under api in the spec. Each key is an engine metric, and its value is the platform metric's short name:
api:
operations:
- type: openai.chat
metrics:
endpoint: /metrics # Optional; defaults to /metrics
map:
'tensorfold:requests_running': requests_running
'tensorfold:time_to_first_token_seconds': ttft_seconds
'engine:kv_used_percent': {to: gpu_cache_usage_percent, scale: 0.01}Rules:
- Use a short name for the target (
ttft_seconds), not the storage namegpuworks_infer_ttft_seconds. The short name must appear in the vocabulary. - Map histograms by family name: Use
tensorfold:time_to_first_token_seconds, not..._bucket. The agent converts_bucket,_sum, and_counttogether. Histograms cannot use label selectors or scaling. Do not separately map lines such as_countfrom the same family. - Select a line by label: A key can include a selector, for example,
'vllm:kv_offload_tiering_chunk_hits_total{tier="1:fs"}': kv_disk_hits_total. Only lines with matching labels are selected; label values inside the selector must be enclosed in double quotes. - Other labels are not retained: Stored series carry only platform labels (instance, deployment, engine, and node). Multiple lines in a counter family (for example, separated by
model_name) are summed. For a gauge family with multiple lines, only the first is used, so use a selector to choose the line you want when a gauge has lines distinguished by labels such as state. - Scaling: The long form
{to: <short-name>, scale: <factor>}applies to counters and gauges. For example, usescale: 0.01to convert a percentage from 0–100 to a ratio from 0–1, orscale: 1000to convert counts in thousands of tokens to tokens. - Inheritance:
inherit: vllmorinherit: sglangfirst applies that engine's complete built-in mappings, then appliesmap. For the same key,maptakes precedence (the template wins). Templates for thevllm/sglangengines inherit their own mappings even withoutinherit. - Once
maporinheritis specified, automatic detection for callable engines no longer runs.
For engines that report only KV pool capacity and free space, map them to kv_capacity_tokens and kv_free_tokens. The platform derives KV utilization (see Derived Metrics in the vocabulary).
3. Import and Fix Validation Errors
When you import YAML, upload a catalog package, or save in the template editor, api.metrics is validated along with the other fields. Any error causes the entire submission to be rejected. The template editor provides completion for target names.
| Error code | Cause |
|---|---|
CATALOG_METRICS_ENDPOINT_INVALID | endpoint is not a path beginning with / |
CATALOG_METRICS_INHERIT_UNKNOWN | inherit is neither vllm nor sglang |
CATALOG_METRICS_SOURCE_INVALID | The key is not a valid metric family name, or the selector syntax is invalid (values require double quotes) |
CATALOG_METRICS_UNKNOWN_METRIC | The target short name is absent from the vocabulary (a typo, or a storage name beginning with gpuworks_infer_) |
CATALOG_METRICS_HISTOGRAM_SELECTOR | The key for a histogram target includes a label selector |
CATALOG_METRICS_HISTOGRAM_SCALE | Scaling is specified for a histogram target |
CATALOG_METRICS_NEGATIVE_SCALE | The scale factor is negative |
CATALOG_METRICS_SHADOWED_HISTOGRAM | A _bucket / _sum / _count line is mapped from a histogram family that is already mapped |
CATALOG_METRICS_DUPLICATE_SOURCE | The same key appears twice (after trimming leading and trailing whitespace) |
CATALOG_METRICS_FORMAT_INVALID | format is neither prometheus nor json |
CATALOG_METRICS_JSON_INHERIT | inherit is specified with format: json |
CATALOG_METRICS_JSON_PATH_INVALID | A key is not a dot-separated key path with format: json |
CATALOG_METRICS_JSON_HISTOGRAM | A mapping targets a histogram with format: json (JSON endpoints support only counters and gauges) |
CATALOG_METRICS_JSON_COMPARE_COUNTER | A comparison expression (path!=value) maps to a counter with format: json |
4. Verify That the Mappings Take Effect
- After saving the template, send an actual inference request through a deployment that references the tier, ensuring that the metrics you want to observe produce samples.
- Wait for the next scrape and check the inference panel in the deployment details page or service monitoring page. You can also query your mapped metrics in the metrics explorer, for example,
gpuworks_infer_requests_running{deployment="<deployment-id>"}. Short requests may finish between scrapes, leaving the running request count at 0; also check whether mapped token or completed request counters increase. - If no samples appear, compare the raw endpoint, path, and mapping names using the troubleshooting table at the end. Saving the template successfully does not by itself confirm that collection works.
New deployments use the current mappings directly. Running deployments need no restart or redeployment: after you save the template, the mappings are pushed to instances that reference the tier. Agents use them on the next scrape after receiving them; offline nodes receive them only after reconnecting. Changes to built-in template mappings from a server upgrade work the same way.
Older agents recognize only simple mappings. Entries with scaling, derived KV utilization, and automatic detection for callable engines take effect only after the node's agent is upgraded.
Examples
Place all the following snippets under the template's api. Preserve existing settings such as api.operations; do not put metrics at the top level of the template.
TensorFold (the built-in GLM-5.3-Flash TensorFold template; names come from the actual /metrics output of TensorFold 0.6.0):
metrics:
map:
'tensorfold:requests_running': requests_running
'tensorfold:requests_waiting': requests_waiting
'tensorfold_health:requests_total': requests_success_total
'tensorfold:prompt_tokens_total': prompt_tokens_total
'tensorfold_health:completion_tokens_total': generation_tokens_total
'tensorfold_health:cached_tokens_total': prompt_tokens_cache_hit_total
'tensorfold:mtp_drafted_total': spec_draft_tokens_total
'tensorfold:mtp_accepted_total': spec_accepted_tokens_total
'tensorfold:time_to_first_token_seconds': ttft_seconds
'tensorfold:request_latency_seconds': e2e_seconds
'tensorfold_health:pool_tokens': kv_capacity_tokens
'tensorfold_health:pool_free_tokens': kv_free_tokensAn adapter that wraps SGLang but changes metric prefixes: Inherit SGLang's mappings, then add the renamed metrics.
metrics:
inherit: sglang
map:
'adapter:num_running_reqs': requests_running
'adapter:num_queue_reqs': requests_waitingA custom engine whose metrics endpoint is not /metrics and that reports KV utilization as a percentage:
metrics:
endpoint: /stats/prometheus
map:
'myengine:active_requests': requests_running
'myengine:pending_requests': requests_waiting
'myengine:output_tokens_total': generation_tokens_total
'myengine:kv_used_percent': {to: gpu_cache_usage_percent, scale: 0.01}JSON Endpoints
Some engines return statistics in a JSON document instead of Prometheus text, such as /metrics in Strata 0.1.38:
{"live": {"state": "generating", "queued": 0, "prompt_tokens": 50, "generated": 1971},
"totals": {"requests": 13, "prompt_tokens": 2548, "output_tokens": 3356, "drafts_offered": 2645, "drafts_accepted": 1578}}Declare format: json under the template's api.metrics and use dot-separated key paths as the keys in map. The following snippet also belongs under api:
metrics:
format: json
map:
'live.state!=idle': requests_running
'live.queued': requests_waiting
'totals.requests': requests_success_total
'totals.prompt_tokens': prompt_tokens_total
'live.prompt_tokens': prompt_tokens_total
'totals.output_tokens': generation_tokens_total
'live.generated': generation_tokens_total
'totals.drafts_offered': spec_draft_tokens_total
'totals.drafts_accepted': spec_accepted_tokens_total- Numeric values at a path are collected; booleans are recorded as 1/0.
nulland missing paths produce no metric. - For strings, write a comparison:
path==valueorpath!=value. A true comparison is recorded as 1, and false as 0 (live.state!=idlerepresents the number of requests being processed). Comparisons can map only to gauges. If the compared value is not a string, no metric is produced. - Only counters and gauges can be mapped. JSON documents contain no histograms, so histogram metrics such as TTFT and end-to-end latency are collected by the gateway.
inheritis not allowed, and built-in engine mappings are not applied. The long form{to, scale}remains available. Two paths mapped to the same metric are summed.- Pay attention to what each counter includes. Strata's
totalsincrements only when requests finish. Mapping only this field makes generation throughput stay unchanged during long responses, then jump all at once at the end.livedescribes the request currently being processed. When it finishes, its generated amount is merged intototals. Mappingtotals.output_tokensandlive.generatedto the same counter sums them, so throughput grows continuously during generation. - A counter summed this way drops when a request is canceled (
liveis cleared, buttotalsis not incremented). The platform treats a counter decrease as an engine restart and counts the entire reading as an increase. To handle this, the agent holds JSON endpoint counters steady: if a decreased value is still at least half the previous reading, it holds the previous reading until the counter exceeds it again. Only a drop below half is passed through as an engine restart.
Troubleshooting
| Symptom | What to check |
|---|---|
| The panel never shows engine data | Whether curl in step 1 returns output; whether endpoint matches the engine's actual path; whether keys match the raw output exactly (including prefixes and colons) |
| Only some metrics have data | Whether the missing metrics exist in the raw output; whether labeled families need selectors |
| Values are off by a fixed factor (such as 100) | The engine uses different units from the vocabulary; add scale |
| A gauge has the wrong value | The family has multiple lines, and only the first was used; select the required line with a selector |
| The throughput chart stays at 0 and spikes when requests finish | The mapped counter increments only when requests finish (check whether # HELP says finished requests). Map a counter that includes in-flight requests instead. For example, use tensorfold_health:completion_tokens_total instead of tensorfold:generation_tokens_total for TensorFold |
Was this page helpful?