tractor/ai/tpt-backends/03_wg_tunnel_bindspace.md

28 KiB
Raw Blame History

Plan 03 — WireGuard (and other tunnels) as a nested bindspace via pyroute2

Tracks gh #482 + the tunnelled-maddr item of #443. Prereq reading: 00_shared_backend_contract.md.

Thesis: WireGuard is not a MsgTransport. It is an interface-layer tunnel that is transparent to socket(2), so the correct abstraction is a bindspace — a scoped, @acm-managed network context that an existing L4 transport (tcp, and later quic/tipc-over-UDP-bearer) binds inside. This plan implements Address.namespace (specd but unused since day one) and the composed/tunnelled maddr grammar, with pyroute2 as the netlink codec and as much of the I/O moved onto trio as the librarys sans-io layer allows.


1. What exists today (verified, per #482)

  • wrap_address() accepts maddr strs (leading-/ dispatch, _addr.py:262). parse_maddr() and mk_maddr() support plain TCP/UDS addresses plus nested, canonical bearer-first /wg/ stacks represented locally as TunnelledAddress wrappers.
  • there is no wg proto in the multiaddr spec yet, but multiformats/py-multiaddr#108 (key form u<base64url>) is merged as of 2026-07-28 (f86519da) — and unreleased, the latest 0.2.0 predating it. Spec registration is still tracked by multiformats/py-multiaddr#107 and gh #483.
  • todays deployable story remains declarative: run wg-quick out-of-band, parse the maddr, strip its wrapper to the overlay (host, port), verify the pubkey against the live tunnel, hand the overlay addr to registry_addrs=/tpt_bind_addrs=. #482 already contains working example code for exactly this.
  • Address.namespace exists in the Protocol (_addr.py:94-101, “the if-available OS-specific network namespace key”). TunnelledAddress implements it from its spec; no concrete transport backend implements it yet.

2. Three layers, three PRs

layer what dep ships
A. declarative commit #482s examples; parse_maddr() learns /wg/u<key> → overlay Address + verified pubkey multiaddr (already), wg(8) CLI first
B. pyroute2 read/verify replace the subprocess.run(['sudo','wg','show']) shelling with netlink queries pyroute2 extra second
C. @acm lifecycle create/configure/tear down wg ifaces + netns from the runtime, as nested bindspaces; implement Address.namespace pyroute2 + CAP_NET_ADMIN third

Each is independently valuable and independently reviewable. Do not attempt C first — the interesting design (nested bindspace @acms) is only well-posed once A has pinned the address grammar and B has proven the netlink path under trio.


3. Layer A — declarative wg maddrs

3.1 the address shape

The decision: a wg segment annotates an existing address, it does not create a new address type. Two candidate encodings; pick (a):

  • (a) TunnelledAddress wrapper (recommended):

    class TunnelledAddress(
        msgspec.Struct,
        frozen=True,
    ):
        overlay: Address         # e.g. TCPAddress
        tunnel: WGTunnelSpec     # proto-specific, frozen

    with .proto_key delegating to overlay.proto_key so every existing table lookup (_addr_to_transport, enable_transports guard at _root.py:391, transport_from_addr()) keeps working untouched, and .unwrap() delegating to overlay.unwrap() so nothing new crosses the wire. .namespace and .bindspace come from the tunnel spec. The wrapper is stripped (→ .overlay) at the moment of bind/connect.

    • ⚠️ is_wrapped_addr() (_addr.py:194) tests type(addr) in _address_types.values() — a bidict of proto_key→type. TunnelledAddress isnt in it and must not be (its not 1:1 with a proto). So either add an explicit isinstance(addr, TunnelledAddress) clause there, or give the wrapper a marker and test structurally. Do the former; its two lines and honest.
    • the reflection in Endpoint.start_listener() (inspect.getmodule(self.addr)) would resolve to the wrappers module, not the transports. So the wrapper must be unwrapped before it reaches Endpoint — i.e. by the bindspace @acm (layer C) or by parse_maddr() (layer A). State this loudly in the docstring; its the #1 way to get this wrong.
    1. add fields to each existing Address type. Rejected: duplicates tunnel logic per-backend and pollutes .unwrap().
class WGTunnelSpec(
    msgspec.Struct,
    frozen=True,
):
    peer_pubkey: str          # std-base64 `wg(8)` form
    iface: str = 'wg0'
    netns: str|None = None
    # layer-C-only fields, unset in layer A
    maybe_endpoint: tuple[str, int]|None = None
    maybe_allowed_ips: tuple[str, ...] = ()

3.2 parse_maddr()/mk_maddr()

Grammar — verified against py-multiaddr#108, first on the baudco/py-multiaddr@wg_support branch and re-verified after it merged upstream (multiformats/py-multiaddr@f86519da); all three forms below parse and round-trip. Note the codec also validates that the key decodes to exactly 32 bytes, so a truncated key is a StringParseError, not a silently-mangled parse:

/ip4/192.168.1.50/udp/51820/wg/u<A_pub>/ip4/10.0.11.1/tcp/1616
\_______ bearer __________/\__ key __/\______ overlay ______/
 underlay, wg `ListenPort`             the `MsgTransport` bind

The /wg/ segment is infix, not suffix — the segments before it are the wg bearer (the underlay (ip, udp-port) that wg(8) itself listens on, per the codec docstrings own /ip4/1.2.3.4/udp/51820/wg/{key} example), and the segments after are the overlay endpoint that tractor binds.

⚠️ CORRECTION — an earlier revision of this plan (and the examples in gh #482) used a suffix form /ip4/10.0.11.1/tcp/1616/wg/u<key>. That parses, but it is semantically inverted: it puts the overlay addr where the bearer belongs, tcp where wgs udp ListenPort goes, and declares no overlay endpoint at all. parse_wg_maddr() in examples/multihost/wg_lan/ now rejects it with an actionable error. Observed protocol-name lists, for writing the match:

maddr [p.name for p in m.protocols()]
/ip4/1.2.3.4/udp/51820/wg/u<k> ['ip4','udp','wg']
/ip4/../udp/../wg/u<k>/ip4/../tcp/.. ['ip4','udp','wg','ip4','tcp']
  • so the three parts have three different owners, and only the third is an Endpoint:

    part socket owner / provisioner runtime role
    bearer kernel-owned; externally provisioned in layer A, tractor bindspace-provisioned in layer C control-plane metadata, never an Endpoint
    /wg/u<key> nothing — its an identity parsed and explicitly verified
    overlay tractors IPCServer application MsgTransport, as .overlay

    This owner-split is the real axis of the design, not whether the maddr stack is “composed” (it is).

  • ⚠️ CORRECTION, an earlier draft of this section specced a hand-rolled _peel_tunnel_segs(proto_names) -> (bearer_names, tunnel_specs, overlay_names). Do not write it. py-multiaddr already ships the whole tunnel compose/peel API and it was simply missed here — see its README “En/decapsulate” and “Tunneling” sections, and gh #443s 2nd bullet which links them. Verified against the pinned rev:

    need API
    isolate the bearer ma.decapsulate_code(P_WG)
    drop the overlay, keep bearer+key ma.decapsulate(overlay_ma)
    per-seg maddrs ma.split()
    rejoin a seg tail Multiaddr.join(*segs)
    read the key ma.value_for_protocol('wg')
    recompose bearer.encapsulate(key).encapsulate(overlay)

    .decapsulate_code() handles the infix /wg/ seg cleanly because it cuts on proto-code and never tries to match an addr value — the key seg has no addr of its own. This is the same NIH trap gh #429 existed to close, one layer up.

  • ⚠️ value_for_protocol('ip4') on a full tunnelled maddr silently returns the first match, i.e. the bearers host. Always call it on a peeled sub-maddr, never the whole stack.

  • parse_maddr() gains a case on [('ip4'|'ip6'), 'udp', 'wg', ('ip4'|'ip6'), <overlay-l4>] → peel w/ the API above, decode the multibase key to std-base64, and return TunnelledAddress(overlay=..., tunnel=WGTunnelSpec( ...)) w/ the bearer recorded in the spec.

  • keep the existing 2-proto cases byte-identical; add the new case after them.

  • nesting (wg-in-wg) falls out of .decapsulate_code() cutting at the last occurrence — peel repeatedly rather than recursing through a bespoke splitter.

  • mk_maddr() inverse for TunnelledAddress is just .encapsulate() composition; dont rebuild strs by hand.

  • pending an upstream release: py-multiaddr#108 is merged, so Multiaddr('/…/wg/u…') parses off a PEP 621 direct-revision pin, since no release carries the codec. Gate parser entry on _wg_proto_code(), implemented as protocols.protocol_with_name('wg') under except ProtocolNotFoundError. Do not probe by parsing a dummy like Multiaddr('/wg/uAAAA') — the codec enforces a 32-byte key, so that raises even when the proto is known. Do not hand-roll a wg parser in tractor — the whole point of #429 was dropping the NIH parser.

3.3 verification helper (pure, composable)

Port #482 §2s pure helpers into tractor/discovery/_tunnel.py, keeping the impure probe cleanly separated until layer B:

def parse_wg_maddr(maddr: str) -> TunnelledAddress: ...   # pure
def wg8_pubkey(multibase_key: str) -> str: ...            # pure
def verify_wg_peer(spec: WGTunnelSpec) -> bool: ...       # layer B

In layer A verify_wg_peer() may shell out (wg show <if> peers), but it must be a single function so layer B swaps only its body. Never call it implicitly from wrap_address()/parse_maddr() — parsing must stay pure and side-effect-free; verification is the callers explicit step (and later, the bindspace @acms).

3.4 deliverables

  • examples/ scripts distilled from #482 §§3-5 (this is the unchecked “commit examples from ^” bullet in #443). They live under examples/multihost/test_docs_examples.py walks examples/ recursively and runs every collected file as a subproc asserting rc == 0 (it doesnt even filter by extension, so a stray README.md would be python-run too), and 'multihost' not in p[0] is already in its exclusion list. Anything needing a real second host or a live tunnel belongs there.
  • a docs/ page: tunnel setup, the maddr form, the two-host run. Keep prose in the docs; keep the examples runnable and minimal.
  • tests: maddr round-trip, TunnelledAddress delegation (proto_key/unwrap identical to overlay), wrap_address() regression (a tunnelled maddr strTunnelledAddress; a plain one → unchanged), and a real end-to-end over a locally-created wg pair gated on CAP_NET_ADMIN (see §5.3).

4. Layer B — pyroute2 under trio

4.1 the library situation (verify at implementation time)

pyroute2 ≥0.9 rewrote its core onto asyncio (AsyncIPRoute; the sync IPRoute wraps it with its own loop). It also ships a WireGuard netlink (generic-netlink) module supporting .set(iface, private_key=..., peer={...}) and .info(iface), plus pyroute2.netns / NetNS for namespaces, and IPRoute.link('add', kind='wireguard', ifname=...).

Three integration options, in increasing trio-nativeness:

  • (1) trio.to_thread.run_sync() around the sync API. Netlink ops here are one-shot, sub-millisecond, and happen at bind/teardown time only — not in the msg hot path. This is the correct default: its ~10 lines, uses a battle-tested API, and costs nothing where its used.
  • (2) sans-io: trio.socket + pyroute2s message codecs. pyroute2s message classes (pyroute2.netlink.rtnl.*, pyroute2.netlink.generic.wireguard.wgmsg) encode/decode independently of its I/O core. So a tractor/ipc/_netlink.py with a small trio NetlinkSocket (trio.socket.socket(AF_NETLINK, SOCK_RAW|SOCK_DGRAM, proto), sendto/recv, seq/pid matching, NLMSG_DONE/NLMSG_ERROR handling) + pyroute2 codecs is very achievable and is the honest reading of “as much trio wrapping as possible where any other async support can be replaced”. Do this for the paths we actually need (link add/del, addr add, wg get/set, netns bind) and only those — a general netlink client is out of scope.
    1. reimplement the codecs. Never.

Recommended split: ship (1) first so layer B is a small, reviewable, behaviour-preserving swap of verify_wg_peer()s body; then land (2) as a follow-up commit for the read path (wg get, link get) where the sans-io surface is smallest, and keep (1) for the privileged mutating ops. Measure before converting anything else — there is no perf argument here, only a “no foreign event loop in a trio actor” argument, which (1) already satisfies (a thread is not an event loop).

Explicitly do not pull in trio-asyncio for pyroute2: it would be the one place in the runtime where an asyncio loop exists for no reason.

4.2 API shape

Pure-ish, functional, @acm for anything with teardown:

async def read_wg_peers(
    iface: str = 'wg0',
    netns: str|None = None,
) -> tuple[str, ...]: ...          # base64 pubkeys

async def read_wg_pubkey(iface: str = 'wg0', ...) -> str: ...

and verify_wg_peer() becomes a thin composition over the two. Note the pure-getter rule: no read_wg_peers(..., create=True).


5. Layer C — nested bindspace @acms + Address.namespace

This is the part #443 and multiaddr_declare_eps.md actually ask for: “for any tunneled maddr-str-entry we deliver a data-structure which can easily be passed to nested @acms which consecutively setup nested net bindspaces for binding the endpoint addrs”.

Layer C is where tractor takes ownership of bindspace orchestration. For a fully bootstrapped deployment it may create the netns and wg iface, configure peers/routes, and ask the kernel to establish the bearers UDP ListenPort through netlink/pyroute2. “Kernel-owned” describes the data-plane socket, not who provisions it: tractor owns the lifecycle while Endpoint/MsgTransport remain responsible only for the overlay application socket.

5.1 the composition

The maddr describes the composed network path and can be used as either a source/listen or destination/dial handle. It does not select the local instance of that network stack. A netns, VRF, interface, user namespace, or equivalent platform resource is orthogonal augmentation carried alongside/below the maddr.

Keep two bindspace representations with deliberately different lifetimes:

class BindspaceSpec(msgspec.Struct, frozen=True):
    '''Serializable spawn/config declaration.'''
    kind: str                 # `netns`, later `vrf`, ...
    key: str|None             # requested name/key, if any


class BindspaceIdentity(msgspec.Struct, frozen=True):
    '''Stable identity of the realized platform resource.'''
    kind: str
    key: str|None
    inode: int|None           # Linux namespace identity


class BindspaceHandle:
    '''Scoped, non-serializable capability for one live bindspace.'''
    spec: BindspaceSpec
    identity: BindspaceIdentity
    namespace_fd: int|None
    ownership: Literal['owned', 'borrowed']


@acm
async def open_bindspace(
    spec: BindspaceSpec,
    *,
    role: Literal['listen', 'dial'],
) -> AsyncGenerator[BindspaceHandle, None]:
    '''
    Provision/borrow one bindspace and yield its live capability.

    '''

The exact field set remains design work; the required split does not: BindspaceSpec crosses config/spawn serialization, while BindspaceHandle contains live OS resources (especially an open namespace FD), pins identity/lifetime, and must never cross msgpack. An FD is a stronger capability than a namespace name: it avoids name-resolution TOCTOU, survives rename/unlink, and identifies the exact namespace the parent provisioned.

open_bindspace() is not an address factory and does not return a TunnelledAddress. At the declaration layer, listener allocation can use the handle to replace an overlay while preserving every tunnel:

async with open_bindspace(
    bindspace_spec,
    role='listen',
) as bindspace:
    listen_decl = declared_addr.get_random(
        bindspace=bindspace,
    )
    transport_addr = strip_tunnels(listen_decl)

That sketch intentionally leaves the .get_random()/bindspace value contract open. A concrete transport call returns a concrete overlay; a declaration-level call may replace the overlay and return a new TunnelledAddress. In either case wrappers remain until the final transport bind/dial boundary, where strip_tunnels() is mandatory.

Per-platform provisioning still composes one resource context per tunnel/bindspace layer:

@acm
async def open_netns(
    spec: BindspaceSpec,
    role: Literal['listen', 'dial'],
) -> AsyncGenerator[BindspaceHandle, None]: ...

@acm
async def open_wg_iface(
    spec: WGTunnelSpec,
    bindspace: BindspaceHandle,
    role: Literal['listen', 'dial'],
) -> AsyncGenerator[WGTunnelSpec, None]: ...

and a driver that folds a list of specs into nested contexts (contextlib.AsyncExitStack for the N-deep case). The parse_endpoints() API (_multiaddr.py:153) is the front door: it already returns dict[name, list[Address|TunnelledAddress]] and the multiaddr_declare_eps.md sketch anticipates the recursive dict[str, list[Address]]|dict[...] return for tunnelled entries. Extend it to carry the tunnel stack, not to enter it.

The caller supplies role; do not infer it from maddr shape. The same composed maddr can name a server source or client destination, and the required local provisioning/ownership differs (§5.3).

5.2 Address.namespace, at last

  • TunnelledAddress.namespace(kind, id) e.g. ('netns', 'tractor-wg0').
  • and the existing backends should implement it as None explicitly (they currently just dont define it), so the Protocol stops lying.
  • consumers to audit: nothing reads .namespace today — so adding it is safe, but the point is that Endpoint/Server.pformat() should start showing it (theres already a # !TODO, always be ns aware! + f'|_netns: {netns}\n' placeholder sitting in Endpoint.pformat(), _server.py:645). Fill that in; its the cheapest possible proof the layer is wired.

Use github/ns_aware@e4688cad as prototype evidence, not code to cherry-pick unchanged. Its /proc/<pid>/ns/<type> inode reader and ip netns identify probe establish the useful (key, inode) identity pair. Layer C should move that shape into BindspaceIdentity, avoid a subprocess where netlink/procfs suffices, and hold the namespace FD in BindspaceHandle to pin the identity.

5.3 the netns/process reality — read this before designing

The headline consequence, stated up front: netns is a runtime-level config API, not an actor-app-code API. It is declared as part of how an actor process is brought up — a spawn-time/boot-time input alongside enable_transports and tpt_bind_addrs — and it is not dynamically re-enterable by app code once the actor is live. There is deliberately no await actor.enter_netns(...). Two hard reasons, both below: setns(2) doesnt retroactively move existing sockets, and its per-thread rather than per-process. Anything that looks like a mid-life API here would be a footgun that silently leaves the IPC server bound in the old namespace.

  • setns(2) with CLONE_NEWNET affects the calling thread only, and sockets already created keep their original netns. A trio actor is effectively single-threaded for our purposes, so “enter the netns, then bind” works — but any to_thread worker (§4.1 option 1!) is in the original netns unless it also setns. Concretely: a wg query issued via trio.to_thread will hit the wrong namespace. Either pass netns= down to pyroute2 (which does the fork/setns dance itself) or pin a dedicated worker. This is the single subtlest bug in this plan — write the test first.
  • entering a netns is process-global-ish and irreversible-ish in practice. Therefore: netns membership belongs to the actor process, decided before the runtime binds, not to a mid-life actor API. Design:
    • the root/parent decides the BindspaceSpec, provisions or borrows it, and passes the spec plus an inherited/transferred namespace-FD capability through the spawn backend (theres already enable_transports/accept_addrs plumbing at _runtime.py:1595-1615 — the netns rides alongside).
    • the child spawn/bootstrap trampoline calls setns() before _runtime.async_main(), IPCServer.listen_on(), parent-channel connection, or creation of any worker thread/socket.
    • only after successful entry does the child drop namespace-entry privileges and initialize the actor runtime.
    • a root/single-actor process follows the same ordering: enter during root bootstrap, never after actor runtime startup.
    • iface/route/WG provisioning is genuinely scoped and remains under the parent/supervisors BindspaceHandle context.
    • document the constraint rather than hiding it; a RuntimeError if namespace entry is attempted after bootstrap.
  • capabilities: iface/netns creation/config needs CAP_NET_ADMIN; entering an existing Linux namespace normally requires CAP_SYS_ADMIN in the owning user namespace. Never sudo from inside the runtime. A privileged parent/helper should provision the stack and open the namespace FD; the child receives only the scoped capability and temporary authority needed to enter it, then drops that authority before actor code runs. This separates create/config authority from enter/use authority and fits user-namespace/capability deployments without granting every actor broad ambient caps. Two supported modes remain:
    1. pre-provisioned out-of-band (layers A/B — the default, and what #482 documents), (ii) runtime-managed when the supervising process/helper holds the required caps. Probe exact required caps and fail loudly with an actionable message otherwise.
  • role semantics are explicit:
    • listen: may create/own the local bindspace, iface, routes, WG peer/listener state, and random local overlay; lifetime normally extends through all listeners and the actor process.
    • dial: may borrow an actor-wide bindspace or ensure local routing and tunnel state reaches the remote stack; it does not own the remote maddr and may need no new local resource at all.
    • source/destination use is an operation property, never permanently encoded into the maddr or inferred from segment ordering.
  • teardown follows capability ownership, not just address type:
    • owned listener bindspaces tear down after endpoints/channels and the actor process have exited;
    • borrowed dial/actor-wide bindspaces only release their handle;
    • nested resources exit inside-out, but shared resources remain until their owning supervisor drops the final capability.
  • teardown must be idempotent and tolerant: an iface/netns already gone must not strand the rest of the teardown — the exact lesson _uds.close_listener()s FileNotFoundError tolerance and _serve_ipc_eps()s per-ep try/except encode. Mirror both.

5.4 tests for layer C

  • unit: fold-N-tunnel-specs-into-nested-@acms, with fakes; assert enter/exit ordering (outermost-last-out) via a trace list.
  • integration, gated on CAP_NET_ADMIN (skip otherwise, and in CI run it in a --cap-add NET_ADMIN container job): create two netns + a wg pair entirely in-process, boot a tractor root in one and a subactor in the other, find_actor() across the tunnel. This is a fantastic test to have and is fully self-contained — no second host, no sudo in the test body.
  • the to_thread-netns-mismatch regression from §5.3, written first (red), then the fix (green), per project convention.
  • bootstrap ordering: assert the child reports the expected namespace inode before parent-channel connect and listener creation.
  • FD capability: rename/unlink the namespace name after opening its FD and prove child entry still selects the pinned inode.
  • privilege drop: prove actor code lacks provisioning caps after entry.
  • role/ownership: fake listen/dial resources and assert owned listener teardown versus borrowed dial-handle release.

6. “Other shuttle-able tpts”

The generalization the #482 follow-up gestures at: once TunnelledAddress + open_bindspace() exist, the same machinery covers any iface-layer tunnel pyroute2 can drive — ipip/gre/sit/vxlan/geneve/bridge/veth. Keep WGTunnelSpec as one frozen struct among a TunnelSpec = WGTunnelSpec|VxlanTunnelSpec|... union with a kind: ClassVar[str], and dispatch open_* by match on it. Design for it now (union + match), implement only wg + netns. veth-pairs-in-netns is the natural second one because it makes the §5.4 integration test possible without wg at all — consider doing it first for exactly that reason.

7. Non-goals

  • no wg userspace implementation, no key exchange, no wg-quick reimplementation (config-file parsing is out of scope; take structured input).
  • no persistence of private keys beyond what layer Cs iface creation needs (and that stays in get_rt_dir(), 0600).
  • macOS/Windows: layers B/C are Linux-only. Layer A (declarative) works anywhere wg does. Gate accordingly and say so in the docs — do not silently no-op.

8. Risks

risk mitigation
to_thread worker runs in the wrong netns §5.3; pass netns= to pyroute2 or pin a worker; test-first
namespace name is renamed/replaced between provision and spawn pass an open namespace FD; verify (key, inode) after child entry
child starts sockets/threads before setns() enter in the spawn bootstrap trampoline before _runtime.async_main(); assert inode ordering
ambient capabilities leak into actor app code split provision/enter authority and drop caps before runtime initialization
dial path tears down a shared actor bindspace encode ownership in BindspaceHandle; borrowed handles never remove resources
py-multiaddr#108 merged but unreleased PEP 621 direct-revision pin + _wg_proto_code() gate; replace with a release floor once published
TunnelledAddress leaks into transport reflection/type dispatch keep wrappers through declaration/bindspace handling, call strip_tunnels() at channel/endpoint boundaries, and retain the boundary regressions
privileged ops in a library never sudo; explicit cap probe + actionable error; pre-provisioned is the default
pyroute2 0.9 asyncio core drags a loop into the actor option (1) is a thread, not a loop; forbid trio-asyncio here (§4.1)
netns teardown strands actor teardown idempotent/tolerant teardown mirroring _uds.close_listener()

9. Follow-up issue seeds

  • veth-in-netns bindspace (unblocks capless-ish integration testing, and is a great local multi-“host” test rig)
  • composed/tunnelled maddr grammar shared with plan 02s /…/quic-v1/… stacks (gh #443)
  • wg proto into the multiaddr spec (gh #483), then flip MsgTransport.maddr to always return Multiaddr (the third #443 bullet)
  • runtime-managed wg key rotation / peer add-remove as a tractor service actor — the natural “actor that owns the network” demo