Skip to main content

Inference Metric Vocabulary

On this page

The platform defines inference metrics only once, in internal/inferencemetrics: short names, types, units, and metrics derived from other metrics. An inference template's api.metrics references these metrics by short name, and imports are validated against this table. Stored series names use gpuworks_infer_<short-name>; for example, ttft_seconds is stored as gpuworks_infer_ttft_seconds. Every series carries the instance, deployment, engine, and node labels.

For metric sources and how to declare them in templates, see Declare Engine Metrics for an Inference Template. For tradeoffs, see ADR 0039.

Types

TypeMeaningMapping Requirements
counterA cumulative value that only increases; panels calculate its rate over a time windowMultiple engine-side lines with the same name (such as _total split by model_name) are summed and stored as a single series
gaugeCurrent value—
histogramA Prometheus histogram family. Raw buckets are not stored: when collected data is written, bucket deltas between consecutive scrapes are used to calculate three quantile gauges, <name>{quantile="0.5"|"0.95"|"0.99"}; <name>_sum and <name>_count are stored separatelyMap by family name (without _bucket); label selectors and scaling are not allowed

Units: requests for request counts, tokens for token counts, ratio for a ratio from 0 to 1 (rendered as a percentage in panels), seconds for seconds, bytes for bytes, and events for event counts.

Metrics

Short NameTypeUnitMeaning
decode_time_secondshistogramsecondsTime a request spends in the decode stage
e2e_secondshistogramsecondsEnd-to-end request duration, as measured by the engine
generation_tokens_totalcountertokensCumulative generated (output) tokens
gpu_cache_usage_percentgaugeratioKV cache utilization; the name retains its historical spelling, but the stored value is a ratio from 0 to 1. Derived from capacity and free space when the engine only reports those values; see below
kv_capacity_tokensgaugetokensKV pool capacity in tokens
kv_disk_hits_totalcountereventsKV offload disk-tier hits
kv_disk_queries_totalcountereventsKV offload disk-tier queries
kv_disk_write_bytes_totalcounterbytesBytes written to disk by KV offload
kv_disk_write_failures_totalcountereventsKV offload disk-write failures
kv_free_tokensgaugetokensFree KV pool capacity in tokens
kv_hit_rate_instantgaugeratioInstantaneous prefix cache hit rate reported directly by the engine
kv_hits_totalcountertokensCumulative prefix cache hit tokens
kv_memory_hits_totalcountereventsKV offload memory-tier hits
kv_memory_queries_totalcountereventsKV offload memory-tier queries
kv_queries_totalcountertokensCumulative prefix cache query tokens
nixl_failed_transfers_totalcountereventsFailed NIXL transfers in PD disaggregation
nixl_kv_expired_totalcountereventsExpired KV requests in PD disaggregation
nixl_transfer_byteshistogrambytesBytes per NIXL transfer in PD disaggregation
nixl_transfer_secondshistogramsecondsDuration of a single NIXL transfer in PD disaggregation
preemptions_totalcountereventsRequest preemptions
prefill_time_secondshistogramsecondsTime a request spends in the prefill stage
prompt_tokens_cache_hit_totalcountertokensCumulative input tokens that hit the local cache
prompt_tokens_external_totalcountertokensCumulative input tokens obtained through external KV transfers
prompt_tokens_local_compute_totalcountertokensCumulative input tokens computed locally
prompt_tokens_totalcountertokensCumulative input tokens
queue_time_secondshistogramsecondsTime requests spend in the queue
requests_runninggaugerequestsRequests currently executing
requests_success_totalcounterrequestsCumulative successfully completed requests
requests_waitinggaugerequestsRequests waiting in the queue
spec_accept_lengthgaugetokensAverage accepted length in speculative decoding
spec_accept_rategaugeratioSpeculative decoding acceptance rate, when reported directly by the engine
spec_accepted_tokens_totalcountertokensCumulative draft tokens accepted in speculative decoding
spec_draft_tokens_totalcountertokensCumulative draft tokens proposed in speculative decoding
tpot_secondshistogramsecondsTime per output token
ttft_secondshistogramsecondsTime to first token, as measured by the engine

Gateway Metrics

The gateway measures each request passing through it independently of the engine's /metrics, so every deployment has gateway metrics, including engines without /metrics (such as Strata).

Stored names use gpuworks_proxy_<short-name>, with the labels deployment and node="server" (requests_total also carries class). Histograms are likewise converted into quantile gauges per deployment for storage.

Short NameTypeUnitMeaning
cached_tokens_totalcountertokensCached input tokens reported by the engine in response usage
completion_tokens_totalcountertokensOutput tokens reported by the engine in response usage
prompt_tokens_totalcountertokensInput tokens reported by the engine in response usage
request_duration_mshistogrammillisecondsTime from handing the request to the instance until the last response byte
requests_inflightgaugerequestsRequests the gateway is currently forwarding
reported_decode_mshistogrammillisecondsThe request's Decode duration reported by the engine in the response (per-request statistic decode_seconds), used as a fallback; see below
reported_prefill_mshistogrammillisecondsThe request's Prefill duration reported by the engine in the response (prefill_seconds), used as a fallback
reported_queue_mshistogrammillisecondsThe request's queue duration reported by the engine in the response (queue_seconds), used as a fallback
requests_totalcounterrequestsRequests forwarded or rejected by the gateway, grouped by class (2xx/4xx/429/5xx/asleep_503)
tpot_mshistogrammillisecondsGateway-measured TPOT: time from the first to the last generated token in each streaming response, divided by the number of output tokens after the first; used as a fallback
ttft_mshistogrammillisecondsTime from handing the request to the instance until the first generated token in the streaming response
usage_missing_totalcounterrequestsResponses received in full but without usage; these are the requests omitted from token counts
wake_hold_wait_mshistogrammillisecondsHow long a request the gateway held while its deployment woke waited, from being held until it was served, refused or abandoned by the client; every held request counts once

Reading and Counting Token Usage

Token counts come only from the engine-reported usage, not from counting streaming chunks: a chunk can contain multiple tokens, and the gateway does not estimate token counts.

  • The protocol registry (inferenceapi.Usage) defines which fields contain usage for each operation. Currently, it supports openai.chat, openai.completions, openai.embeddings, and systemone.decide. A decision generates no text and records only usage.input_tokens: the output_tokens the engine replies with counts answered questions, not generated tokens, so it is not counted as output tokens.
  • Completion streams carry text in a choice's text field (there is no delta), which is used for first-token timing.
  • The gateway reads only the final 4 KiB of a response; usage must be in the response's last event or JSON object.
  • Usage is not read from compressed responses (non-identity Content-Encoding) or responses that are neither JSON nor SSE.

The gateway automatically requests usage only for vLLM / SGLang streaming text requests: the endpoint is /v1/chat/completions or /v1/completions, the request body is JSON the gateway can read, stream: true is set, and the client has not specified stream_options.

When these conditions are met, the gateway adds include_usage on behalf of the client, then removes the extra usage chunk the engine sends as a result. The client receives the same bytes it would receive without this addition (ADR 0042). Requests for which the client specifies stream_options are forwarded unchanged.

This request parameter is not added for callable engines. TensorFold and Strata always include usage. Other responses without usage count toward usage_missing_total.

When a Response Counts as Complete

Responses truncated by a client disconnect count toward neither tokens nor usage_missing_total.

If the last response event (finish_reason, along with usage in the same event or after it) has already reached the gateway, this is not considered an interrupted response: if the client closes the connection after receiving the last event and leaves only data: [DONE] unread, the response is still complete. Tokens are counted normally, and the request record also records a normal completion.

Cache Hit Counts

Cache hit counts come from usage.prompt_tokens_details.cached_tokens. When the field is absent, the following rules apply:

  • When SGLang has cache reporting enabled (--enable-cache-report), it omits prompt_tokens_details when there are 0 hits; for these deployments, the gateway reads an absent value as 0.
  • When vLLM has --enable-prompt-tokens-details enabled, it includes the field even when there are 0 hits.
  • In all other cases, an absent field means unknown; the request is excluded from the "Uncached input × TTFT" distribution.

How Gateway TTFT Is Measured

Gateway TTFT (time to first token) differs from the engine's ttft_seconds measurement: the gateway starts timing when it hands the request to the instance, including queue and network time.

The first token is the first nonempty content, reasoning_content, reasoning, or tool_calls in a streaming response. The gateway reads the entire stream line by line as SSE; individual lines larger than 64 KiB are excluded. TTFT is not recorded for non-streaming responses.

Fallback Statistics for Missing Engine Metrics

tpot_ms and the three reported_*_ms metrics supplement the engine's tpot_seconds, queue_time_seconds, prefill_time_seconds, and decode_time_seconds, respectively. The gateway records a fallback histogram only when the engine's built-in mapping lacks the corresponding metric (inferencemetrics.GatewayFallbacksFor):

  • vLLM exports all four itself; the gateway records none of them.
  • SGLang exports TPOT and queue time, so the gateway records only Prefill and Decode. Currently, SGLang responses do not include per-request statistics, so these two are empty in practice.
  • For callable engines (TensorFold, Strata, and others), the gateway records all four.

tpot_ms requires a streaming response with usage and at least 2 output tokens. It measures from the first generated token observed by the gateway to the last text event, including network jitter. When the engine emits multiple tokens at once (speculative decoding), the time is still averaged by token count.

reported_*_ms aggregates engine-reported values in responses (per-request statistics) into per-deployment distributions. If a response does not include a value, nothing is recorded.

Per-Request Statistics

Per-request statistics are timings the engine reports for each request in its response. When the response ends, the gateway reads them and writes them into the request record (ux.db; see View Metrics and Search Logs). These statistics serve two purposes:

  • The request view uses them to break completed requests into queue, Prefill, and Decode stages.
  • When the engine does not export stage histograms itself, the gateway aggregates them into reported_*_ms histograms for the "Stage duration" chart.

The fields read are determined by the engine's built-in declarations or the template's api.request_stats.

Short NameUnitMeaning
queue_secondssecondsTime spent waiting for scheduling after the request reaches the engine
prefill_secondssecondsTime from being scheduled to producing the first token
decode_secondssecondsTime from the first token to the last token

Starting with vLLM 0.25.0, --enable-per-request-metrics enables all three. TensorFold and Strata report Prefill and Decode, but not queue time.

Derived Metrics

After each scrape, the agent calculates metrics not directly reported by the engine according to the following table. If the engine reports the target metric itself, the engine's value takes precedence.

TargetInputsCalculation
gpu_cache_usage_percentkv_capacity_tokens, kv_free_tokens(capacity − free) / capacity; no value is produced if capacity ≤ 0 or free capacity is outside [0, capacity]

Queued Request Counts and Per-Request Queue Duration

requests_waiting (for example, Strata's /metrics field live.queued) counts queued requests at the sampling instant. It is not a millisecond duration. Neither this gauge nor a service histogram can populate an individual request's engine_queue_ms. That field comes only from queue_seconds reported in the same response. Missing fields and explicit null mean unknown; an explicit 0 means measured zero queue time. If queue timing is available without Prefill timing, retain the known queue segment and leave the remainder as unsplit waiting.

Unmodified Strata 0.1.38 standard responses contain only timings.prompt_ms and timings.predicted_ms, so queue duration remains unknown. Waiting may also include preprocessing and network time; renaming all waiting to engine queueing would be inaccurate. The updated built-in Strata catalog carries a source-hash-checked runtime patch that reports initial-generation admission waiting as timings.queue_ms in the same response. It excludes model loading and later MCP rounds without enabling body monitoring. Update the deployment and recreate old instances to emit the field. For pinned source evidence and integration constraints, see Observability Limits of Per-Request Engine Queue Time.

Was this page helpful?