Architecture Decision Records
ADR 0024: Do Not Fall Back to the Control Plane When the Data Lane Disconnects
Date: 2026-09-24 · Status: Accepted · Supersedes the "data lane disconnects independently" item in §3.2 of artifacts/archive/2026-09-08-tunnel-bandwidth/plan.md
The original dual-lane tunnel plan (the predecessor to this ADR: artifacts/archive/2026-09-08-tunnel-bandwidth/plan.md) falls back to the ctrl session for data-plane stream kinds such as tunnel/logtail/model.*/image.* while the data lane is disconnected, with "in-flight streams failing (the proxy returns 502, consistent with current disconnection behavior)." This fallback defeats the isolation purpose of the dual-lane design itself: the agent's data plane reconnects independently with backoff capped at 30 seconds. During this window, all inference proxy, log tailing, and image/model transfer traffic crowds onto the ctrl session, slowing heartbeats, status reports, and web terminals. A data-plane failure thus escalates into a control-plane failure; in the worst case, heartbeat timeouts incorrectly mark online nodes as offline. A subtler issue is a detection race: when TCP has disconnected but yamux has not yet detected it through keepalive or its read loop (IsClosed() still returns false), the old logic still attempts to open new streams on this seemingly healthy but dead session. Header-write failures propagate directly upward as 502 responses rather than triggering any form of retry.
The data-lane state of Session is now explicitly divided into three states: none (never attached—older agents, --data-lane=false, and server_dial, for which a single lane is the normal, permanent form), attached (currently healthy), and detached (previously attached, now disconnected). The none state retains the original behavior: data-plane kinds still use ctrl because these nodes only have one physical connection. The attached state uses the data lane directly. The detached state no longer falls back to ctrl: the caller waits within its own context for the data lane to reattach, up to a fixed constant (hub.dataLaneReattachWait, 5 seconds). This covers one round trip in the common case where the agent "disconnects after a successful attach, then retries immediately for its first reconnect without backoff." If reattachment does not occur in time, the typed sentinel hub.ErrDataLaneUnavailable is returned, and the caller treats it as a connection-level failure.
Recovery from the detection race applies only when the session is confirmed dead: either OpenStream fails and the session's IsClosed() is now true (go-yamux v5 always sets its internal shutdown flag before closing shutdownCh and returning a shutdown-induced error to the caller, so checking IsClosed() immediately after the call neither misses a closed session nor falsely identifies one), or the stream opens but writing its header fails (the yamux send path is fire-and-forget, and write failure is the only available signal on this path, so it is unconditionally treated as session death). Only these two cases mark the session detached and call Close() on it, then wait for reconnection and retry, without ever using ctrl. When OpenStream fails because of context cancellation/timeouts, resource-scope limits, exhausted stream IDs, or similar causes, the session itself remains healthy. The error is returned unchanged to the caller, with no detach, retry, or close. An early implementation unconditionally detached on any OpenStream/header-write failure, escalating a single request cancellation into node-wide data-plane unavailability (the lane being awaited still existed but was incorrectly considered disconnected, and would not be replaced until it actually disconnected). This is corrected.
If the data lane dies after a stream opens but before the request receives a response (2026-10-09 addendum), it is likewise reported as "this node's data channel is unavailable." Streams opened on the data lane (hub.laneStream) check their owning yamux session's IsClosed() on a read/write error; if it is true, they wrap the error in ErrDataLaneUnavailable and immediately detach that lane. The determination is reliable: go-yamux v5 sets the shutdown flag before forcibly closing all streams in the session with stream reset: connection closed. If the agent resets only an individual stream (because dialing the engine fails or the engine disconnects), the session stays alive, the original error is preserved, and the proxy still returns 502. Thus, when the proxy encounters this type of failure before response headers, it returns a retryable 503 (with Retry-After), just as when the lane is already disconnected at stream opening. With multiple replicas, connection-level retries switch to another replica. A disconnection after response headers have been sent cannot change the status code; the client sees a truncated response.
The tradeoff is explicit: during the detached window, requests to single-replica deployments wait up to 5 seconds and then receive a retryable 503 (with Retry-After), rather than an immediate 502. For multi-replica deployments, the inference gateway's connection-level retry (ADR 0023) switches to another replica as a fallback, usually without a noticeable effect on users. This cost is considered far smaller than the risk of escalating a data-plane failure into a control-plane failure, especially because uncontrolled control-plane latency can trigger more severe cascading effects such as incorrectly marking nodes offline or declaring running instances unreachable.
Out of scope: no new persistent data structures or wire protocol changes (the DataHello/SessionID/stale_session rejection logic stays unchanged). file.large_read (OpenLargeFileRead) retains its stricter existing semantics: it requires the lane to be attached, and immediately rejects both none and detached without waiting. This stream's contract is already "no lane means no fetch, and the caller immediately takes another path"; waiting another 5 seconds would only delay that fallback path.
Later: the metric and log batches the agent ships are routed by the same rule; see ADR 0064. Also, a server_dial node gets a second lane dialed by the server when the agent enables the data lane, so it is not permanently in the none state.
Was this page helpful?