Architecture Decision Records
ADR 0066: Agent state is grouped by owner and lifetime, and the composition root only wires
On this page
Date: 2026-11-15 · Status: Accepted
Context
internal/agent/app is one package of 89 files and 15.2k lines (non-test); NodeRuntime has 77 fields (counting those inside anonymous structs), 12 mutexes and 233 methods, and OnRPC, with the helper switch it hands over to, dispatches 51 RPCs. Images, model files, card health and doctor all hang off this struct and share its locks: a change to any feature changes it, and no subsystem can be tested or replaced on its own.
internal/agent/engine keeps an instance's state in 12 parallel maps keyed by UUID, and each exit path deletes some of them. The four that describe the engine process (sleep level, prewarm, L1 host buffer, processes on its cards) are cleared only when the container exits by itself: not when the server removes or stops the instance, and not when a dead container is replaced directly by the next sync. Measured (GWKS-321): a new container under the same UUID was reported holding the old process's L1 host buffer, so the server skipped its host memory check; the new process was not prewarmed; and when an instance desired asleep got a new process, the agent took the old process's sleep for the new one, so the new engine stayed awake with all its memory while the server believed it asleep.
The server's controller was already reorganized this way: one self-locking type per kind of memory (GWKS-304), a test that makes every field say whether it is dropped per deployment, and another that really writes, forgets and asserts nothing is left (GWKS-303). This ADR makes that the agent's pattern.
Decision
- Group by owner and lifetime. State under one key (an instance UUID, a model file, an image reference, …) that ends at one moment goes into one record (a
struct), held by onemap[key]*record, instead of a parallel map per fact. An instance's state is split into records by lifetime: the desired generation, one reconcile attempt (in-flight, pending, cancel), and one engine process (processFacts). - A record is bounded by what it describes. The engine process's record starts empty where a container is created and is dropped whole where the container is confirmed gone (
reclaimInstance) and on a runtime switch, never piecemeal on each exit path. The empty start covers every replacement that does not pass throughreclaimInstance(a dead container replaced directly by a sync, a runtime switch). A new fact goes into the record it belongs to and is cleared with it. Code that writes the record runs inside that UUID's reconcile attempt (the inFlight serialization); a removal drops it only after the attempt ends, so no old attempt writes it back. - Every field is classified, and per-instance cleanup is proven by a test.
TestEveryEngineFieldIsClassifiedlists eachEnginefield: per-instance ones say whether the server removing the instance drops them; the others say whether they are a dependency, configuration, a lock or a global counter; a map keyed by string can only be per-instance.TestRemovingAnInstanceLeavesNothingUnderItsUUIDmakes a sleeping instance leave entries in every table, removes it, and asserts no field marked as dropped still holds it. This is the same pattern as the controller'sTestEveryControllerFieldIsClassifiedwithTestForgetDeploymentLeavesNoMemory. A new field fails until it is classified, or until it is cleared if marked as dropped. - Subsystems own their state and locks, behind narrow dependencies. Images, model files, card health and doctor each become a package or type holding their own fields and locks, given only the interfaces they need (driver, reporter, storage guard, …), testable from a zero value or a constructor.
NodeRuntimeis the composition root: one field per subsystem, plus the connection and lifecycle wiring. - Shared state belongs to the composition root, and locks do not cross subsystems. Configuration several subsystems read (the network configuration
netcfg, the models directory, registry auth and the puller) is held by the composition root and handed to subsystems as a read-only getter or snapshot; no subsystem holds another's configuration lock. Subsystems do not take each other's locks: to look up another subsystem's table (the peer relay looks up both model syncs and image syncs) a subsystem calls a narrow method the other exposes, which answers under its own lock. No code holds two subsystems' locks at once. - RPC is a table, not a switch.
OnRPCbecomes a "method name → handler" table whose handlers the subsystems provide;NodeRuntimeonly looks them up and handles the shared errors. - A ratchet keeps it so.
agent_node_runtime_fields,agent_node_runtime_mutexes,agent_node_runtime_methods,agent_onrpc_casesandagent_engine_struct_mapsinscripts/arch-constraints.txtonly go down (initially 77 / 12 / 233 / 51 / 9); loosening one needs a# raised:reason. They are measured byscripts/structcounton the syntax tree: several names on one line count per name; pointer locks,sync.Mapand map types of the package count; fields inside an anonymous struct still count (wrapping is not leaving); RPCsOnRPChands to a helper switch count. Only moving into a named subsystem type counts as leaving. The method count guards behaviour: fields moved away while their methods stay onNodeRuntimeand touch the subsystem's state directly is not a split.
Consequences
- First step (GWKS-321): the engine process's four maps become
processes map[string]*processFacts, starting empty when a container is created and dropped whole inreclaimInstanceand on a runtime switch;Enginemap fields go 12 → 9. A new container under the same UUID no longer inherits the old process's facts:TestANewContainerUnderTheSameUUIDKnowsNothingOfTheOldProcess(the server removing or stopping it) andTestAReplacedEngineProcessIsPutToSleepAgain(a dead container replaced by a sync, a runtime switch: the new process really is put to sleep again). - The subsystems are then extracted one at a time: images (the peer relay reads the image syncs through a narrow method of the image subsystem) → model files → card health and doctor → the RPC table, each merged on its own, tightening the ratchet as it goes.
Costs and what is not done
runtimeEpochis kept per UUID for the agent process's life on purpose: a removal moves it to a fresh global value, so a reused UUID never matches an old attempt. The cost is oneuint64per UUID seen.- The desired-generation and reconcile-attempt maps (
desired,inFlight,inFlightCancel,pending,cleanupRequired) are not merged in this step: they are maintained together by reconcile's serialization protocol, and merging them means changing that protocol with them, which is a change of its own. NodeRuntimegets no field classification test in this step: most of its fields are subsystem state that leaves whole as each subsystem is extracted, guarded by the ratchet; the per-instance cleanup test is required onEnginefirst.- Splitting subsystems does not change the agent–server protocol or behaviour.
Was this page helpful?