28 KiB
Plan 03 — WireGuard (and other tunnels) as a nested bindspace via pyroute2
Tracks gh #482 + the tunnelled-maddr item of #443. Prereq reading: 00_shared_backend_contract.md.
Thesis: WireGuard is not a MsgTransport. It is an interface-layer tunnel that is transparent to socket(2), so the correct abstraction is a bindspace — a scoped, @acm-managed network context that an existing L4 transport (tcp, and later quic/tipc-over-UDP-bearer) binds inside. This plan implements Address.namespace (spec’d but unused since day one) and the composed/tunnelled maddr grammar, with pyroute2 as the netlink codec and as much of the I/O moved onto trio as the library’s sans-io layer allows.
1. What exists today (verified, per #482)
wrap_address()accepts maddrstrs (leading-/dispatch,_addr.py:262).parse_maddr()andmk_maddr()support plain TCP/UDS addresses plus nested, canonical bearer-first/wg/stacks represented locally asTunnelledAddresswrappers.- there is no
wgproto in the multiaddr spec yet, but multiformats/py-multiaddr#108 (key formu<base64url>) is merged as of 2026-07-28 (f86519da) — and unreleased, the latest0.2.0predating it. Spec registration is still tracked by multiformats/py-multiaddr#107 and gh #483. - today’s deployable story remains declarative: run
wg-quickout-of-band, parse the maddr, strip its wrapper to the overlay(host, port), verify the pubkey against the live tunnel, hand the overlay addr toregistry_addrs=/tpt_bind_addrs=. #482 already contains working example code for exactly this. Address.namespaceexists in the Protocol (_addr.py:94-101, “the if-available OS-specific network namespace key”).TunnelledAddressimplements it from its spec; no concrete transport backend implements it yet.
2. Three layers, three PRs
| layer | what | dep | ships |
|---|---|---|---|
| A. declarative | commit #482’s examples; parse_maddr() learns /wg/u<key> → overlay Address + verified pubkey |
multiaddr (already), wg(8) CLI |
first |
B. pyroute2 read/verify |
replace the subprocess.run(['sudo','wg','show']) shelling with netlink queries |
pyroute2 extra |
second |
C. @acm lifecycle |
create/configure/tear down wg ifaces + netns from the runtime, as nested bindspaces; implement Address.namespace |
pyroute2 + CAP_NET_ADMIN |
third |
Each is independently valuable and independently reviewable. Do not attempt C first — the interesting design (nested bindspace @acms) is only well-posed once A has pinned the address grammar and B has proven the netlink path under trio.
3. Layer A — declarative wg maddrs
3.1 the address shape
The decision: a wg segment annotates an existing address, it does not create a new address type. Two candidate encodings; pick (a):
(a)
TunnelledAddresswrapper (recommended):class TunnelledAddress( msgspec.Struct, frozen=True, ): overlay: Address # e.g. TCPAddress tunnel: WGTunnelSpec # proto-specific, frozenwith
.proto_keydelegating tooverlay.proto_keyso every existing table lookup (_addr_to_transport,enable_transportsguard at_root.py:391,transport_from_addr()) keeps working untouched, and.unwrap()delegating tooverlay.unwrap()so nothing new crosses the wire..namespaceand.bindspacecome from the tunnel spec. The wrapper is stripped (→ .overlay) at the moment of bind/connect.- ⚠️
is_wrapped_addr()(_addr.py:194) teststype(addr) in _address_types.values()— abidictof proto_key→type.TunnelledAddressisn’t in it and must not be (it’s not 1:1 with a proto). So either add an explicitisinstance(addr, TunnelledAddress)clause there, or give the wrapper a marker and test structurally. Do the former; it’s two lines and honest. - the reflection in
Endpoint.start_listener()(inspect.getmodule(self.addr)) would resolve to the wrapper’s module, not the transport’s. So the wrapper must be unwrapped before it reachesEndpoint— i.e. by the bindspace@acm(layer C) or byparse_maddr()(layer A). State this loudly in the docstring; it’s the #1 way to get this wrong.
- ⚠️
- add fields to each existing
Addresstype. Rejected: duplicates tunnel logic per-backend and pollutes.unwrap().
- add fields to each existing
class WGTunnelSpec(
msgspec.Struct,
frozen=True,
):
peer_pubkey: str # std-base64 `wg(8)` form
iface: str = 'wg0'
netns: str|None = None
# layer-C-only fields, unset in layer A
maybe_endpoint: tuple[str, int]|None = None
maybe_allowed_ips: tuple[str, ...] = ()3.2 parse_maddr()/mk_maddr()
Grammar — verified against py-multiaddr#108, first on the baudco/py-multiaddr@wg_support branch and re-verified after it merged upstream (multiformats/py-multiaddr@f86519da); all three forms below parse and round-trip. Note the codec also validates that the key decodes to exactly 32 bytes, so a truncated key is a StringParseError, not a silently-mangled parse:
/ip4/192.168.1.50/udp/51820/wg/u<A_pub>/ip4/10.0.11.1/tcp/1616
\_______ bearer __________/\__ key __/\______ overlay ______/
underlay, wg `ListenPort` the `MsgTransport` bind
The /wg/ segment is infix, not suffix — the segments before it are the wg bearer (the underlay (ip, udp-port) that wg(8) itself listens on, per the codec docstring’s own /ip4/1.2.3.4/udp/51820/wg/{key} example), and the segments after are the overlay endpoint that tractor binds.
⚠️ CORRECTION — an earlier revision of this plan (and the examples in gh #482) used a suffix form /ip4/10.0.11.1/tcp/1616/wg/u<key>. That parses, but it is semantically inverted: it puts the overlay addr where the bearer belongs, tcp where wg’s udp ListenPort goes, and declares no overlay endpoint at all. parse_wg_maddr() in examples/multihost/wg_lan/ now rejects it with an actionable error. Observed protocol-name lists, for writing the match:
| maddr | [p.name for p in m.protocols()] |
|---|---|
/ip4/1.2.3.4/udp/51820/wg/u<k> |
['ip4','udp','wg'] |
/ip4/../udp/../wg/u<k>/ip4/../tcp/.. |
['ip4','udp','wg','ip4','tcp'] |
so the three parts have three different owners, and only the third is an
Endpoint:part socket owner / provisioner runtime role bearer kernel-owned; externally provisioned in layer A, tractor bindspace-provisioned in layer C control-plane metadata, never an Endpoint/wg/u<key>nothing — it’s an identity parsed and explicitly verified overlay tractor’sIPCServerapplication MsgTransport, as.overlayThis owner-split is the real axis of the design, not whether the maddr stack is “composed” (it is).
⚠️ CORRECTION, an earlier draft of this section specced a hand-rolled
_peel_tunnel_segs(proto_names) -> (bearer_names, tunnel_specs, overlay_names). Do not write it.py-multiaddralready ships the whole tunnel compose/peel API and it was simply missed here — see its README “En/decapsulate” and “Tunneling” sections, and gh #443’s 2nd bullet which links them. Verified against the pinned rev:need API isolate the bearer ma.decapsulate_code(P_WG)drop the overlay, keep bearer+key ma.decapsulate(overlay_ma)per-seg maddrs ma.split()rejoin a seg tail Multiaddr.join(*segs)read the key ma.value_for_protocol('wg')recompose bearer.encapsulate(key).encapsulate(overlay).decapsulate_code()handles the infix/wg/seg cleanly because it cuts on proto-code and never tries to match an addr value — the key seg has no addr of its own. This is the same NIH trap gh #429 existed to close, one layer up.⚠️
value_for_protocol('ip4')on a full tunnelled maddr silently returns the first match, i.e. the bearer’s host. Always call it on a peeled sub-maddr, never the whole stack.parse_maddr()gains a case on[('ip4'|'ip6'), 'udp', 'wg', ('ip4'|'ip6'), <overlay-l4>]→ peel w/ the API above, decode the multibase key to std-base64, and returnTunnelledAddress(overlay=..., tunnel=WGTunnelSpec( ...))w/ the bearer recorded in the spec.keep the existing 2-proto cases byte-identical; add the new case after them.
nesting (wg-in-wg) falls out of
.decapsulate_code()cutting at the last occurrence — peel repeatedly rather than recursing through a bespoke splitter.mk_maddr()inverse forTunnelledAddressis just.encapsulate()composition; don’t rebuildstrs by hand.pending an upstream release: py-multiaddr#108 is merged, so
Multiaddr('/…/wg/u…')parses off a PEP 621 direct-revision pin, since no release carries the codec. Gate parser entry on_wg_proto_code(), implemented asprotocols.protocol_with_name('wg')underexcept ProtocolNotFoundError. Do not probe by parsing a dummy likeMultiaddr('/wg/uAAAA')— the codec enforces a 32-byte key, so that raises even when the proto is known. Do not hand-roll awgparser intractor— the whole point of #429 was dropping the NIH parser.
3.3 verification helper (pure, composable)
Port #482 §2’s pure helpers into tractor/discovery/_tunnel.py, keeping the impure probe cleanly separated until layer B:
def parse_wg_maddr(maddr: str) -> TunnelledAddress: ... # pure
def wg8_pubkey(multibase_key: str) -> str: ... # pure
def verify_wg_peer(spec: WGTunnelSpec) -> bool: ... # layer BIn layer A verify_wg_peer() may shell out (wg show <if> peers), but it must be a single function so layer B swaps only its body. Never call it implicitly from wrap_address()/parse_maddr() — parsing must stay pure and side-effect-free; verification is the caller’s explicit step (and later, the bindspace @acm’s).
3.4 deliverables
examples/scripts distilled from #482 §§3-5 (this is the unchecked “commit examples from ^” bullet in #443). They live underexamples/multihost/—test_docs_examples.pywalksexamples/recursively and runs every collected file as a subproc assertingrc == 0(it doesn’t even filter by extension, so a strayREADME.mdwould bepython-run too), and'multihost' not in p[0]is already in its exclusion list. Anything needing a real second host or a live tunnel belongs there.- a
docs/page: tunnel setup, the maddr form, the two-host run. Keep prose in the docs; keep the examples runnable and minimal. - tests: maddr round-trip,
TunnelledAddressdelegation (proto_key/unwrapidentical to overlay),wrap_address()regression (a tunnelled maddrstr→TunnelledAddress; a plain one → unchanged), and a real end-to-end over a locally-created wg pair gated onCAP_NET_ADMIN(see §5.3).
4. Layer B — pyroute2 under trio
4.1 the library situation (verify at implementation time)
pyroute2 ≥0.9 rewrote its core onto asyncio (AsyncIPRoute; the sync IPRoute wraps it with its own loop). It also ships a WireGuard netlink (generic-netlink) module supporting .set(iface, private_key=..., peer={...}) and .info(iface), plus pyroute2.netns / NetNS for namespaces, and IPRoute.link('add', kind='wireguard', ifname=...).
Three integration options, in increasing trio-nativeness:
- (1)
trio.to_thread.run_sync()around the sync API. Netlink ops here are one-shot, sub-millisecond, and happen at bind/teardown time only — not in the msg hot path. This is the correct default: it’s ~10 lines, uses a battle-tested API, and costs nothing where it’s used. - (2) sans-io:
trio.socket+ pyroute2’s message codecs.pyroute2’s message classes (pyroute2.netlink.rtnl.*,pyroute2.netlink.generic.wireguard.wgmsg) encode/decode independently of its I/O core. So atractor/ipc/_netlink.pywith a small trioNetlinkSocket(trio.socket.socket(AF_NETLINK, SOCK_RAW|SOCK_DGRAM, proto),sendto/recv, seq/pid matching,NLMSG_DONE/NLMSG_ERRORhandling) + pyroute2 codecs is very achievable and is the honest reading of “as much trio wrapping as possible where any other async support can be replaced”. Do this for the paths we actually need (link add/del, addr add, wg get/set, netns bind) and only those — a general netlink client is out of scope. - reimplement the codecs. Never.
Recommended split: ship (1) first so layer B is a small, reviewable, behaviour-preserving swap of verify_wg_peer()’s body; then land (2) as a follow-up commit for the read path (wg get, link get) where the sans-io surface is smallest, and keep (1) for the privileged mutating ops. Measure before converting anything else — there is no perf argument here, only a “no foreign event loop in a trio actor” argument, which (1) already satisfies (a thread is not an event loop).
Explicitly do not pull in trio-asyncio for pyroute2: it would be the one place in the runtime where an asyncio loop exists for no reason.
4.2 API shape
Pure-ish, functional, @acm for anything with teardown:
async def read_wg_peers(
iface: str = 'wg0',
netns: str|None = None,
) -> tuple[str, ...]: ... # base64 pubkeys
async def read_wg_pubkey(iface: str = 'wg0', ...) -> str: ...and verify_wg_peer() becomes a thin composition over the two. Note the pure-getter rule: no read_wg_peers(..., create=True).
5. Layer C — nested bindspace @acms + Address.namespace
This is the part #443 and multiaddr_declare_eps.md actually ask for: “for any tunneled maddr-str-entry we deliver a data-structure which can easily be passed to nested @acms which consecutively setup nested net bindspaces for binding the endpoint addrs”.
Layer C is where tractor takes ownership of bindspace orchestration. For a fully bootstrapped deployment it may create the netns and wg iface, configure peers/routes, and ask the kernel to establish the bearer’s UDP ListenPort through netlink/pyroute2. “Kernel-owned” describes the data-plane socket, not who provisions it: tractor owns the lifecycle while Endpoint/MsgTransport remain responsible only for the overlay application socket.
5.1 the composition
The maddr describes the composed network path and can be used as either a source/listen or destination/dial handle. It does not select the local instance of that network stack. A netns, VRF, interface, user namespace, or equivalent platform resource is orthogonal augmentation carried alongside/below the maddr.
Keep two bindspace representations with deliberately different lifetimes:
class BindspaceSpec(msgspec.Struct, frozen=True):
'''Serializable spawn/config declaration.'''
kind: str # `netns`, later `vrf`, ...
key: str|None # requested name/key, if any
class BindspaceIdentity(msgspec.Struct, frozen=True):
'''Stable identity of the realized platform resource.'''
kind: str
key: str|None
inode: int|None # Linux namespace identity
class BindspaceHandle:
'''Scoped, non-serializable capability for one live bindspace.'''
spec: BindspaceSpec
identity: BindspaceIdentity
namespace_fd: int|None
ownership: Literal['owned', 'borrowed']
@acm
async def open_bindspace(
spec: BindspaceSpec,
*,
role: Literal['listen', 'dial'],
) -> AsyncGenerator[BindspaceHandle, None]:
'''
Provision/borrow one bindspace and yield its live capability.
'''The exact field set remains design work; the required split does not: BindspaceSpec crosses config/spawn serialization, while BindspaceHandle contains live OS resources (especially an open namespace FD), pins identity/lifetime, and must never cross msgpack. An FD is a stronger capability than a namespace name: it avoids name-resolution TOCTOU, survives rename/unlink, and identifies the exact namespace the parent provisioned.
open_bindspace() is not an address factory and does not return a TunnelledAddress. At the declaration layer, listener allocation can use the handle to replace an overlay while preserving every tunnel:
async with open_bindspace(
bindspace_spec,
role='listen',
) as bindspace:
listen_decl = declared_addr.get_random(
bindspace=bindspace,
)
transport_addr = strip_tunnels(listen_decl)That sketch intentionally leaves the .get_random()/bindspace value contract open. A concrete transport call returns a concrete overlay; a declaration-level call may replace the overlay and return a new TunnelledAddress. In either case wrappers remain until the final transport bind/dial boundary, where strip_tunnels() is mandatory.
Per-platform provisioning still composes one resource context per tunnel/bindspace layer:
@acm
async def open_netns(
spec: BindspaceSpec,
role: Literal['listen', 'dial'],
) -> AsyncGenerator[BindspaceHandle, None]: ...
@acm
async def open_wg_iface(
spec: WGTunnelSpec,
bindspace: BindspaceHandle,
role: Literal['listen', 'dial'],
) -> AsyncGenerator[WGTunnelSpec, None]: ...and a driver that folds a list of specs into nested contexts (contextlib.AsyncExitStack for the N-deep case). The parse_endpoints() API (_multiaddr.py:153) is the front door: it already returns dict[name, list[Address|TunnelledAddress]] and the multiaddr_declare_eps.md sketch anticipates the recursive dict[str, list[Address]]|dict[...] return for tunnelled entries. Extend it to carry the tunnel stack, not to enter it.
The caller supplies role; do not infer it from maddr shape. The same composed maddr can name a server source or client destination, and the required local provisioning/ownership differs (§5.3).
5.2 Address.namespace, at last
TunnelledAddress.namespace→(kind, id)e.g.('netns', 'tractor-wg0').- and the existing backends should implement it as
Noneexplicitly (they currently just don’t define it), so the Protocol stops lying. - consumers to audit: nothing reads
.namespacetoday — so adding it is safe, but the point is thatEndpoint/Server.pformat()should start showing it (there’s already a# !TODO, always be ns aware!+f'|_netns: {netns}\n'placeholder sitting inEndpoint.pformat(),_server.py:645). Fill that in; it’s the cheapest possible proof the layer is wired.
Use github/ns_aware@e4688cad as prototype evidence, not code to cherry-pick unchanged. Its /proc/<pid>/ns/<type> inode reader and ip netns identify probe establish the useful (key, inode) identity pair. Layer C should move that shape into BindspaceIdentity, avoid a subprocess where netlink/procfs suffices, and hold the namespace FD in BindspaceHandle to pin the identity.
5.3 the netns/process reality — read this before designing
The headline consequence, stated up front: netns is a runtime-level config API, not an actor-app-code API. It is declared as part of how an actor process is brought up — a spawn-time/boot-time input alongside enable_transports and tpt_bind_addrs — and it is not dynamically re-enterable by app code once the actor is live. There is deliberately no await actor.enter_netns(...). Two hard reasons, both below: setns(2) doesn’t retroactively move existing sockets, and it’s per-thread rather than per-process. Anything that looks like a mid-life API here would be a footgun that silently leaves the IPC server bound in the old namespace.
setns(2)withCLONE_NEWNETaffects the calling thread only, and sockets already created keep their original netns. A trio actor is effectively single-threaded for our purposes, so “enter the netns, then bind” works — but anyto_threadworker (§4.1 option 1!) is in the original netns unless it alsosetns. Concretely: a wg query issued viatrio.to_threadwill hit the wrong namespace. Either passnetns=down topyroute2(which does the fork/setns dance itself) or pin a dedicated worker. This is the single subtlest bug in this plan — write the test first.- entering a netns is process-global-ish and irreversible-ish in practice. Therefore: netns membership belongs to the actor process, decided before the runtime binds, not to a mid-life actor API. Design:
- the root/parent decides the
BindspaceSpec, provisions or borrows it, and passes the spec plus an inherited/transferred namespace-FD capability through the spawn backend (there’s alreadyenable_transports/accept_addrsplumbing at_runtime.py:1595-1615— the netns rides alongside). - the child spawn/bootstrap trampoline calls
setns()before_runtime.async_main(),IPCServer.listen_on(), parent-channel connection, or creation of any worker thread/socket. - only after successful entry does the child drop namespace-entry privileges and initialize the actor runtime.
- a root/single-actor process follows the same ordering: enter during root bootstrap, never after actor runtime startup.
- iface/route/WG provisioning is genuinely scoped and remains under the parent/supervisor’s
BindspaceHandlecontext. - document the constraint rather than hiding it; a
RuntimeErrorif namespace entry is attempted after bootstrap.
- the root/parent decides the
- capabilities: iface/netns creation/config needs
CAP_NET_ADMIN; entering an existing Linux namespace normally requiresCAP_SYS_ADMINin the owning user namespace. Neversudofrom inside the runtime. A privileged parent/helper should provision the stack and open the namespace FD; the child receives only the scoped capability and temporary authority needed to enter it, then drops that authority before actor code runs. This separates create/config authority from enter/use authority and fits user-namespace/capability deployments without granting every actor broad ambient caps. Two supported modes remain:- pre-provisioned out-of-band (layers A/B — the default, and what #482 documents), (ii) runtime-managed when the supervising process/helper holds the required caps. Probe exact required caps and fail loudly with an actionable message otherwise.
- role semantics are explicit:
listen: may create/own the local bindspace, iface, routes, WG peer/listener state, and random local overlay; lifetime normally extends through all listeners and the actor process.dial: may borrow an actor-wide bindspace or ensure local routing and tunnel state reaches the remote stack; it does not own the remote maddr and may need no new local resource at all.- source/destination use is an operation property, never permanently encoded into the maddr or inferred from segment ordering.
- teardown follows capability ownership, not just address type:
- owned listener bindspaces tear down after endpoints/channels and the actor process have exited;
- borrowed dial/actor-wide bindspaces only release their handle;
- nested resources exit inside-out, but shared resources remain until their owning supervisor drops the final capability.
- teardown must be idempotent and tolerant: an iface/netns already gone must not strand the rest of the teardown — the exact lesson
_uds.close_listener()’sFileNotFoundErrortolerance and_serve_ipc_eps()’s per-eptry/exceptencode. Mirror both.
5.4 tests for layer C
- unit: fold-N-tunnel-specs-into-nested-
@acms, with fakes; assert enter/exit ordering (outermost-last-out) via a trace list. - integration, gated on
CAP_NET_ADMIN(skip otherwise, and in CI run it in a--cap-add NET_ADMINcontainer job): create two netns + a wg pair entirely in-process, boot atractorroot in one and a subactor in the other,find_actor()across the tunnel. This is a fantastic test to have and is fully self-contained — no second host, nosudoin the test body. - the
to_thread-netns-mismatch regression from §5.3, written first (red), then the fix (green), per project convention. - bootstrap ordering: assert the child reports the expected namespace inode before parent-channel connect and listener creation.
- FD capability: rename/unlink the namespace name after opening its FD and prove child entry still selects the pinned inode.
- privilege drop: prove actor code lacks provisioning caps after entry.
- role/ownership: fake listen/dial resources and assert owned listener teardown versus borrowed dial-handle release.
6. “Other shuttle-able tpts”
The generalization the #482 follow-up gestures at: once TunnelledAddress + open_bindspace() exist, the same machinery covers any iface-layer tunnel pyroute2 can drive — ipip/gre/sit/vxlan/geneve/bridge/veth. Keep WGTunnelSpec as one frozen struct among a TunnelSpec = WGTunnelSpec|VxlanTunnelSpec|... union with a kind: ClassVar[str], and dispatch open_* by match on it. Design for it now (union + match), implement only wg + netns. veth-pairs-in-netns is the natural second one because it makes the §5.4 integration test possible without wg at all — consider doing it first for exactly that reason.
7. Non-goals
- no wg userspace implementation, no key exchange, no
wg-quickreimplementation (config-file parsing is out of scope; take structured input). - no persistence of private keys beyond what layer C’s iface creation needs (and that stays in
get_rt_dir(), 0600). - macOS/Windows: layers B/C are Linux-only. Layer A (declarative) works anywhere
wgdoes. Gate accordingly and say so in the docs — do not silently no-op.
8. Risks
| risk | mitigation |
|---|---|
to_thread worker runs in the wrong netns |
§5.3; pass netns= to pyroute2 or pin a worker; test-first |
| namespace name is renamed/replaced between provision and spawn | pass an open namespace FD; verify (key, inode) after child entry |
child starts sockets/threads before setns() |
enter in the spawn bootstrap trampoline before _runtime.async_main(); assert inode ordering |
| ambient capabilities leak into actor app code | split provision/enter authority and drop caps before runtime initialization |
| dial path tears down a shared actor bindspace | encode ownership in BindspaceHandle; borrowed handles never remove resources |
| py-multiaddr#108 merged but unreleased | PEP 621 direct-revision pin + _wg_proto_code() gate; replace with a release floor once published |
TunnelledAddress leaks into transport reflection/type dispatch |
keep wrappers through declaration/bindspace handling, call strip_tunnels() at channel/endpoint boundaries, and retain the boundary regressions |
| privileged ops in a library | never sudo; explicit cap probe + actionable error; pre-provisioned is the default |
| pyroute2 0.9 asyncio core drags a loop into the actor | option (1) is a thread, not a loop; forbid trio-asyncio here (§4.1) |
| netns teardown strands actor teardown | idempotent/tolerant teardown mirroring _uds.close_listener() |
9. Follow-up issue seeds
veth-in-netns bindspace (unblocks capless-ish integration testing, and is a great local multi-“host” test rig)- composed/tunnelled maddr grammar shared with plan 02’s
/…/quic-v1/…stacks (gh #443) wgproto into the multiaddr spec (gh #483), then flipMsgTransport.maddrto always returnMultiaddr(the third #443 bullet)- runtime-managed wg key rotation / peer add-remove as a
tractorservice actor — the natural “actor that owns the network” demo