Architecture Decision Records
ADR 0064: Metric and Log Batches Ride a Telemetry Stream on the Data Lane, and Logs Are Buffered on the Agent Within a Bound
On this page
Date: 2026-11-13 · Status: Accepted (supplements ADR 0024)
Background
Metric batches (metrics) and the log batches of containers, the node and the agent (logs) used to be JSON envelopes on the ctrl stream, sharing one yamux stream and the agent's write lock with heartbeats and RPC responses. The ctrl lane has a fixed 256 KiB window, so an RPC response could queue behind up to one window of logs. Measured at 40 ms RTT with a node's two lanes sharing one link (TestNoisyNodeCtrlLatency): a node that logs heavily had an RPC round-trip p50 of 954 ms at 2 Mbps against 58 ms for a quiet node, and 145 ms against 45 ms at 10 Mbps.
While the agent had no connection it dropped the batches it had to send. Measured (TestInstanceLogsWrittenWhileTheServerIsDownReachTheLogCenter): with the server down for 5 seconds, about 120 lines the instance wrote during that time and before the node came back online were lost for good, with no sign of a gap in the log center (GWKS-258).
proto also declared a metrics stream kind that no code ever opened; it has been removed.
Decision
- Telemetry stream. A new stream kind,
telemetry, is opened by the agent on the data lane and carriesmetricsandlogsenvelopes (JSON Lines, the same format as on ctrl). There is one per batch type on each data lane, its header'srefnamingmetricsorlogs, opened on first use and replaced along with the lane. Two streams keep a server waiting for log ingest room (item 5) from holding metrics back; the monitoring pages and the gateway's load signals depend on metrics. The server accepts no other agent-opened kind on the data lane and requires one of those tworefvalues, resetting anything else; a new stream of a batch type is read only after the previous one's lines, so batches are ingested in the order sent. It does not act on other envelopes on a telemetry stream, and it resets a stream whose line exceeds 128 MiB. - Capability. The server advertises
telemetry.streamin its own hello. The agent uses the telemetry stream only when the server advertises it; an older server keeps receiving the batches on ctrl, as before. - A route per connection, no ctrl fallback. After the handshake the agent knows whether the connection will have a data lane (the server issued a session ID, and either the agent dials the data lane or, with
server_dial, the server dials it back). If it will not (an older server,--data-lane=false), batches go on ctrl. If it will, they wait for the data lane and then ride the telemetry stream; until then the send returnsconn.ErrTelemetryUnavailablerather than going on ctrl first, which is exactly when the backlog to resend after a reconnect is largest. If the data lane has not attached within 30 seconds (for example, the network passes only ctrl), the connection's telemetry goes on ctrl until the lane attaches, then moves to it, once. At that moment the last batches on ctrl and the first on the telemetry stream are read by different server goroutines, so log lines with the same millisecond timestamp may swap places (logs are ordered by timestamp, so that is the whole effect). Falling back to ctrl logs an agent warning (data lane did not attach; telemetry uses ctrl until it does), searchable in the log center under sourceagent. Once the data lane is chosen and later goes down, the send returnsconn.ErrTelemetryUnavailableinstead of moving onto ctrl, as in ADR 0024. - Bounded log buffer. The agent puts log batches into an in-memory FIFO (bounded at 16 MiB, estimated as each line's text plus a fixed overhead) and sends them in order whenever it can. When it cannot (no connection, data lane not yet attached or down) it keeps them, and retries on reconnect (
Bind, in the background so the connection's own heartbeat and read loops are not held up) and every 2 seconds; each pass sends only the batches queued when it began, so no writer drains an endless stream of others' logs. When the buffer is full it drops the oldest batches, never the one being sent, and never a batch merely because it was just added (a single batch larger than the bound still goes out whole unless newer batches arrive before it does; the bound is for the backlog an outage builds up); the dropped lines are summarized, per instance and source, in onewarnmarker line (gpuworks: N log lines from <start> to <end> were dropped: …) sent ahead of the surviving logs, so the log center shows the gap. While a send is blocked on the link, later log writers wait behind it, so pressure still reaches the container through thedocker logs -fpipe without dropping more lines. - The server no longer drops logs from the telemetry stream. Logs on ctrl are dropped in whole batches, counted, when the server's ingest queue is full (the ctrl read loop must not wait). The telemetry stream belongs to this agent alone, so the server waits when its ingest queue is full; the wait reaches the agent through yamux flow control, and the logs stay in the agent's buffer, which drops them with a marker only when full, as in item 4. The server takes only complete, newline-terminated lines: after a write error (for example, the 30-second write timeout while the server waits) the agent closes, not resets, the stream and resends the batch on a new one; closing lets the server still read the complete lines it received, and the cut-off last line does not count as delivered. When the session ends, a batch already read and waiting for ingest room gets 10 more seconds.
- Metrics are not buffered. Metrics stay best-effort and are dropped, without a warning, while there is no connection or the data lane is not attached: stale samples are of no use to the monitoring pages, and they would compete with logs for the buffer.
Result
Under the same measurement, the heavily logging node's RPC p50 is 72 ms at 2 Mbps (quiet node 57 ms) and 48 ms at 10 Mbps (45 ms). The lines an instance writes while the server is down for 5 seconds all arrive after the reconnect, with no gap in their sequence.
Costs and Non-Goals
- The buffer is in memory: an agent process restart loses logs not yet sent. Durability and at-least-once delivery need an on-disk journal and server receipts; that is a separate design (an implementation is in progress on the HA branch).
- No receipts: a batch counts as sent once written to the stream (into yamux's send buffer). If the data lane drops right after, the server dies after receiving the batch but before storing it, or the session ends while the server is still waiting for ingest room for it and none comes within 10 seconds, that batch, and any queued behind it in the server's receive window, is still lost (the last case is recorded only in the server's drop count). All three need receipts to rule out, which this does not do.
- At the start of each connection, metrics produced before the data lane attaches (usually under a second) are dropped.
- This buffer and
benchPending(the short list of pending benchmark statuses) stay separate: that one counts entries, this one counts bytes and records drops, so their needs differ. - No extra rate limit on instance logs: with logs off ctrl, a heavily logging node no longer delays its own RPCs, which removes the need for one (GWKS-259).
- Single-lane nodes (an older server,
--data-lane=false) still send telemetry on ctrl, with the latency they had; the buffer applies to them as well.
Was this page helpful?