tractor/ai/tpt-backends/02_quic_iroh_backend.md

Plan 02 — QUIC backend via iroh FFI, uniffi-async rewritten onto trio

Tracks gh #353. Prereq reading: 00_shared_backend_contract.md.

Thesis: the value of iroh over “just QUIC” is NodeId-addressed, NAT-traversing, relay-fallback endpoints — i.e. a tractor actor tree that spans hosts without a reachable listening socket. The cost is that iroh’s python surface is uniffi-generated asyncio and its listener is not a socket. This plan spends its complexity budget in exactly two places: a trio-native uniffi future bridge, and a trio.abc.Listener/Stream adapter pair. Everything else is contract boilerplate.

---

1. Library selection (decided, with the rejected alternatives)

Chosen: iroh (PyPI, from n0-computer/iroh-ffi), pinned to a single minor. The iroh python package is a uniffi binding over the rust iroh crate (QUIC via quinn/noq).

Rejected, and why — record these so the next implementer doesn’t relitigate:

• aioquic (sans-io + asyncio): genuinely trio-portable (hypercorn already pairs its sans-io core with a trio UDP server, see the links in #353) and dependency-light. But it gives us only QUIC — no NodeId identity, no hole punching, no relay. We’d be reimplementing iroh’s whole reason for existing. Keep as the documented fallback if the FFI bridge (§2) proves unmaintainable; the MsgTransport and Listener adapters from §3 are ~90% reusable against an aioquic core, which is a deliberate design property of this plan.
• quiche / quinn via a hand-rolled PyO3 ext: strictly more work than reusing iroh-ffi, and puts us in the build-wheels business.
• trio-asyncio: viable shortcut to run the asyncio-shaped bindings under trio, and tractor already ships infected-asyncio machinery (tractor.to_asyncio, tests/test_infected_asyncio.py). Rejected as the primary design because it makes every IPC send/recv cross a loop-boundary shim in the hot path, and because #353 asks explicitly for the asyncio support to be “rewritten for trio”. But: build it first as the throwaway spike (§6 step 0) to de-risk the iroh API surface before writing the bridge.

Version pinning: iroh moves fast and has had breaking API renames across minors. Pin iroh>=X.Y,<X.Y+1 in a quic extra, and write down the exact resolved version + the generated iroh/_uniffi* module layout in the module docstring, because §2 depends on generated-code internals.

Step 0 of implementation is an API-truth pass: install the pinned iroh, python -c "import iroh; help(iroh)", and record in this doc’s §1.1 the real names of: endpoint builder, secret key type, connect/accept, bi-stream open/accept, the send/recv methods and their exact signatures/return types, and whether they’re async def. Everything below uses provisional names and must be reconciled. Do not skip this; do not guess from memory.

1.1 API-truth table (fill in during step 0)

concept	provisional name	actual (fill in)
secret key	iroh.SecretKey.generate()
endpoint builder	iroh.Endpoint.builder(...).bind()
node id	endpoint.node_id() -> str
node addr (relay + direct)	iroh.NodeAddr
dial	await endpoint.connect(node_addr, alpn)
accept conn	await endpoint.accept()
open bi-stream	await conn.open_bi()
accept bi-stream	await conn.accept_bi()
send	await send_stream.write_all(b)
recv	await recv_stream.read(n) -> bytes\|None
half-close	await send_stream.finish()

---

2. The trio-native uniffi future bridge (tractor/ipc/_uniffi_trio.py)

2.1 what uniffi actually generates

uniffi’s async support does not use asyncio semantically — it uses asyncio only as the executor for a poll loop. The generated python for an async fn is, in shape:

1. call _uniffi_..._<method>(...) → returns an opaque RustFuture handle (a void*/u64).
2. loop: call ffi_..._rust_future_poll_<T>(handle, callback, callback_data). The callback is a C-ABI fn pointer invoked from an arbitrary rust thread with a poll-result code (READY/MAYBE_READY).
3. the generated glue’s callback resolves an asyncio.Future via loop.call_soon_threadsafe(...); the coroutine awaits it, then re-polls.
4. on ready: ffi_..._rust_future_complete_<T>(handle, &call_status) → the value; then ffi_..._rust_future_free_<T>(handle).

The asyncio dependency is confined to step 3. That is the whole insight: the bridge is ~40 lines.

2.2 the trio version

async def await_rust_future(
    poll: Callable,      # ffi_..._rust_future_poll_<T>
    complete: Callable,  # ffi_..._rust_future_complete_<T>
    free: Callable,      # ffi_..._rust_future_free_<T>
    handle: int,
    lift: Callable[[Any], Any],
) -> Any:
    '''
    Drive a `uniffi` rust-future to completion on the current
    `trio` task, bridging rust-thread wakeups via
    `TrioToken.run_sync_soon()`.

    '''
    token = trio.lowlevel.current_trio_token()
    while True:
        wake = trio.Event()
        # NOTE, invoked from a *rust* thread!
        def _cb(_data, poll_code):
            token.run_sync_soon(wake.set)

        cb = _UNIFFI_FUTURE_CALLBACK(_cb)   # keep a strong ref!
        poll(handle, cb, 0)
        await wake.wait()
        if <poll_code was READY>:
            break
    try:
        status = _UniffiRustCallStatus.default()
        res = complete(handle, status)
        _uniffi_check_call_status(status)   # reuse generated helper
        return lift(res)
    finally:
        free(handle)

Critical details, each a real bug if missed:

• token.run_sync_soon() is the only trio API callable from a foreign thread, and it is documented as such. Use it; do not use trio.from_thread.run_sync (requires a trio thread context) and do not touch the Event directly from the callback.
• the poll code must reach the trio side. Capture it in a nonlocal/1-slot list written by the callback before run_sync_soon, since the callback owns the value. Handle MAYBE_READY by re-polling (the loop above does).
• keep the ctypes callback object alive across the await — a GC’d CFUNCTYPE trampoline is a segfault. Bind it to a local and make sure the local outlives the poll() call window.
• cancellation. await wake.wait() is a trio checkpoint, so a Cancelled can fire while rust still owns the future. On cancel we must still free(handle) — and per uniffi, the correct sequence is to call the generated ffi_..._rust_future_cancel_<T>(handle) then continue polling to completion before free. Wrap the whole thing so the cancel path does: with trio.CancelScope(shield=True): cancel(handle); <drain poll loop>; free(handle). Bounded shield (add a trio.move_on_after() with a module-level constant) so a wedged rust future can’t make an actor un-cancellable — tractor is SC-first and an unbounded shield here would violate that.
• trio.lowlevel.current_trio_token() must be captured on the trio side (not in the callback).

2.3 how to apply it to the generated bindings

Do not fork/vendor the generated iroh python. Instead ship a narrow re-dispatch shim:

• write tractor/ipc/_uniffi_trio.py with await_rust_future() plus a @cm patch_uniffi_for_trio() that monkey-patches the generated module’s single async-driver entrypoint (in current uniffi that’s _uniffi_rust_call_async / _rust_call_async, one function) to the trio implementation.
• verify at import time that the expected symbol exists and raise a clear, actionable error naming the pinned iroh version if not. A silent fallback to asyncio would be a nightmare to debug.
• plan for this to break on iroh/uniffi upgrades. Mitigate with (a) a unit test that drives one trivial iroh async call under bare trio.run() and asserts no event loop was ever created (asyncio.get_event_loop_policy() untouched / asyncio._get_running_loop() is None), and (b) a docstring pointing at the uniffi codegen template this mirrors.

If step 0 reveals the generated code is structurally hostile to this (e.g. asyncio imported and used at module scope for more than the driver), fall back to option (b): run iroh under tractor.to_asyncio infected mode and open the follow-up to revisit. Say so in the PR rather than fighting it.

---

3. Mapping QUIC onto MsgTransport

3.1 the layering decision

QUIC natively multiplexes streams inside one connection. The mapping that preserves all existing tractor semantics with the least new code:

iroh Endpoint      ==  one per actor (process)     -> the "listener"
iroh Connection    ==  one per peer actor           -> pooled
iroh bi-stream     ==  one `Channel`/`MsgTransport`  -> 1:1

• keep the 4-byte <I length-prefix framing unchanged. It’s redundant-ish over a QUIC stream but it means MsgpackTransport is reused verbatim, and framing is cheap. Revisit only after it works.
• one-task-per-stream falls out naturally, which is exactly the #353 note about QUIC sub-stream QoS/cancellation fitting trio.
• layer_key: int = 4 still (QUIC is L4-ish); note in a comment that this backend is really 4+security+multiplex.

Connection pooling is the one place we add state the other backends don’t have: dialing the same peer twice should reuse the Connection and open a second bi-stream. Implement as a module-level dict[NodeId, Connection] guarded by a trio.Lock… no — that’s a per-process cache with lifetime/teardown hazards. Instead reuse the codebase’s existing idiom: tractor.trionics.maybe_open_context() keyed on the node-id, which already solves exactly this (one-cached-resource- per-key, refcounted, teardown-on-last-exit) and whose teardown semantics were just hardened (gh #488). Use it; do not hand-roll a cache. Anything concurrency-subtle here should get the conc-anal skill run over it.

3.2 IrohAddress

class IrohAddress(
    msgspec.Struct,
    frozen=True,
):
    _node_id: str                 # 32B ed25519 pubkey, hex or z32
    _alpn: str = 'tractor/0'      # the bindspace!
    # optional dial hints; NOT part of identity
    maybe_relay_url: str|None = None
    maybe_direct_addrs: tuple[str, ...] = ()

    proto_key: ClassVar[str] = 'iroh'   # ?or 'quic'; see §3.2.1
    unwrapped_type: ClassVar[type] = tuple[str, str]
    def_bindspace: ClassVar[str] = 'tractor/0'

• .unwrap() -> (node_id_str, alpn_str) — a (str, str) tuple, which is unambiguously distinct from TCPAddress’s (str, int). But careful: wrap_address()’s UDS case is case (_, filename) if type(filename) is str — which already catches (str, str). So the iroh case MUST be ordered before the UDS case and guarded, e.g. case (str() as nid, str() as alpn) if _is_node_id(nid): with _is_node_id() a cheap length+alphabet check. Add a regression test asserting a UDS (dir, filename) pair still wraps to UDSAddress — this is the exact “wrong transport loaded” hazard _addr.py:214 warns about.
• .bindspace → self._alpn. This is the honest analogue: the ALPN is the set of endpoints willing to talk to you, and two tractor deployments sharing an iroh network are separated by ALPN exactly as two UDS deployments are separated by directory. Include a tractor version/proto epoch in the default ALPN so incompatible runtimes can’t handshake.
• .is_valid → node-id parses, alpn non-empty.
• get_root() is the hard one. There is no well-known-port analogue: an iroh node id is a keypair, so “the host’s default registrar addr” requires a persisted secret key. Design:
  • the root/registrar’s secret key lives at get_rt_dir() / 'iroh_registrar.key' (0600), created on first use.
  • get_root() must stay pure and import-time-safe (contract §2.3: _default_lo_addrs is built at import!). So get_root() reads the key file if present and otherwise returns an IrohAddress with _node_id=''/sentinel, and the generation happens in an explicit sibling — ensure_registrar_key() -> IrohAddress — called from the listen path. Pure getter, explicit setter; do not smuggle key generation into get_root().
  • this almost certainly means _default_lo_addrs must become lazy for this backend. Land that refactor as its own prep commit (a default_lo_addrs() that computes per-call instead of the import-time dict) — it also unblocks plan 03’s netns-scoped defaults.
• get_random(): generate a fresh SecretKey per subactor and return its node-id. Note this runs post-fork pre-listen (contract §4) and costs an ed25519 keygen (~µs, fine). The secret can’t live in a frozen Address, so it must be stashed where the listen path can find it: a module-level dict[node_id, SecretKey] populated by get_random() and consumed+popped by start_listener(). Ugly but honest; document it and note the alternative (thread the key through Endpoint) as a follow-up.

3.2.1 proto_key: 'iroh' vs 'quic'

Use 'quic' for the proto_key/--tpt-proto name and name the module _quic.py, with iroh as the implementation. Rationale: it keeps the door open for the aioquic fallback (§1) without a user-visible rename, and it matches how uds is a proto name rather than a lib name. Put iroh-specific bits behind an internal _iroh submodule if the file gets big.

3.3 the trio.abc adapters — where the real work is

Contract §3 says a non-socket backend needs three upstream generalizations. Land them as a prep PR, before any iroh code, so they can be reviewed on their own merits with tcp/uds still the only backends:

1. Endpoint.start_listener() must not assume .socket.getsockname(). Use the same Address.rebind_from_sockname: ClassVar[bool] gate that plan 01 §3.2 introduces — coordinate so it lands once. (If plan 01 lands first, this is free.)
2. transport_from_stream() (_types.py:92) must not assume trio.SocketStream. Replace the sock.family match with: check isinstance(stream, trio.SocketStream) → existing family match; else look for a stream.tpt_key: ClassVar[MsgTransportKey] attribute on the adapter and use it. Keeps the existing path byte-identical and makes new stream types self-describing (a much better shape than growing an isinstance ladder).
3. type annotations: handle_stream_from_peer(stream: trio.SocketStream) → trio.abc.Stream; Endpoint._listener: SocketListener|None → trio.abc.Listener|None; MsgTransport.stream: trio.SocketStream → trio.abc.Stream. Annotation-only, zero behaviour change.

Then the adapters:

class QuicMsgStream(trio.abc.HalfCloseableStream):
    '''
    A single `iroh` bi-directional QUIC stream presented as
    a `trio` byte-stream so `MsgpackTransport` can frame over
    it unmodified.

    '''
    tpt_key: ClassVar[MsgTransportKey] = ('msgpack', 'quic')

    def __init__(self, conn, send, recv) -> None: ...
    async def send_all(self, data: bytes) -> None: ...
    async def wait_send_all_might_not_block(self) -> None: ...
    async def receive_some(self, max_bytes: int|None = None) -> bytes: ...
    async def send_eof(self) -> None: ...
    async def aclose(self) -> None: ...

Non-negotiable behaviours (each maps to a match case that already exists in _transport.py and must keep working):

• receive_some() returns b'' at clean EOF → MsgpackTransport._iter_packets() sees header == b'' and raises TransportClosed(loglevel='transport'). This is the graceful-disconnect path the whole runtime relies on; get it right first.
• a reset/aborted stream → raise trio.BrokenResourceError.
• use after local close → raise trio.ClosedResourceError (ideally with 'another task closed this fd'-equivalent text absent, so the raise_on_report branch at _transport.py:290 stays quiet).
• send_all() on a closed peer → trio.BrokenResourceError.
• honour trio’s one-task-per-direction rule: guard with trio._util.ConflictDetector equivalents (or just document + assert), because MsgpackTransport already serializes sends with a StrictFIFOLock but recvs are single-task by construction.
• buffering: if iroh’s read() doesn’t support “read up to n”, receive_some() must maintain an internal leftover buffer. Note MsgpackTransport wraps us in tricycle.BufferedReceiveStream anyway, so receive_some() just needs some nonzero-progress contract.

class QuicListener(trio.abc.Listener):
    '''
    Accepts iroh `Connection`s and yields one `QuicMsgStream`
    per accepted bi-stream, so `trio.serve_listeners()` spawns
    one `handle_stream_from_peer()` per `Channel`.

    '''
    async def accept(self) -> QuicMsgStream: ...
    async def aclose(self) -> None: ...

The accept-side subtlety: trio.abc.Listener.accept() yields one stream per call, but iroh gives us connections which then yield streams. So QuicListener needs an internal trio.MemoryReceiveChannel[QuicMsgStream] fed by a background task-pair (one task accepting connections, one per connection accepting bi-streams). trio.abc.Listener has no nursery, so: make the listener constructed by an @acm that owns the nursery, and have start_listener() be that @acm’s driver.

⚠️ this collides with Endpoint.start_listener() being a plain async def returning a listener. Two options: - (a) hang the nursery off the Endpoint’s existing listen_tn — _serve_ipc_eps() already creates listen_tn and passes it into every Endpoint (_server.py:1063-1074), and Endpoint.listen_tn is right there. So start_listener() can self.listen_tn.start_soon(...) the acceptor tasks. Recommended: no upstream signature change, correct lifetime (dies with the ep group), and it’s why listen_tn is on the struct in the first place. - (b) change start_listener() to a @acm. Bigger blast radius; only if (a) proves insufficient.

Since start_listener() is called via inspect.getmodule(addr) with only addr= (contract §1.3), option (a) needs the Endpoint itself. Either add ep= to the module-level start_listener() call signature (all backends ignore it except quic → small upstream change, do it as part of the prep PR and make it keyword-only with a default) or have QuicListener.accept() lazily spawn via trio.lowlevel.current_task().parent_nursery (rejected — fragile, implicit). Do the explicit ep= kwarg.

3.4 maddr

Multiaddr already standardizes the pieces:

/ip4/<h>/udp/<p>/quic-v1                     # direct
/ip4/<h>/udp/<p>/quic-v1/p2p/<node-id>       # direct + identity
/dns/<relay-host>/tcp/443/tls/ws/p2p/<node>  # relay-ish

• primary form: /p2p/<node-id> alone is a legal maddr and is the only required component for iroh dialling — relay + direct addrs are discovery hints. So mk_maddr() emits /p2p/<node_id> and, when known, prefixes the direct /ip4/../udp/../quic-v1/.
• /p2p/ values are multihash-encoded peer ids; an iroh node-id is a raw ed25519 key. Converting requires the identity multihash + libp2p key protobuf wrapper. Decide: emit the raw node-id under a tractor-local /iroh/<node-id> segment (needs upstream registration, same track as wg/tipc, gh #483) rather than pretending to be a libp2p peer-id we can’t round-trip. Return the str form until upstream lands (MsgTransport.maddr is Multiaddr|str).
• this backend is the strongest argument for gh #443’s tunnelled/composed maddr item: /ip4/../udp/../quic-v1/.. is a composed stack. Cross-reference plan 03 §5 so the two grammars land compatibly.

---

4. Discovery integration

• iroh’s node-id addressing means the tractor registrar can hold IrohAddresses that are reachable from anywhere with no port-forwarding — that is the headline feature. The registrar itself works unchanged.
• iroh has its own discovery (DNS/pkarr/mdns). Out of scope; note in the follow-up that tractor.discovery could eventually delegate to it, which would be the direct analogue of plan 01’s TIPC-topology idea.
• relay servers: default to n0’s public relays for the demo, document self-hosting (docs.iroh.computer’s dedicated-infra page is linked from #353), and make the relay set a start_listener() kwarg.

5. Security note

QUIC is TLS-1.3-always and iroh authenticates by node-id, so this backend is the first tractor transport with real transport security and peer authentication. Two things follow: 1. an allowlist hook — an actor should be able to reject inbound connections from unknown node-ids before the Aid handshake. Natural home: a predicate kwarg on start_listener(), evaluated in QuicListener’s connection acceptor task. Sketch it; ship it in PR 1 if cheap (it is). 2. do not claim any security property for the other backends by association. tcp/uds/tipc remain unauthenticated; that’s what plan 03 (wg) is for.

6. Commit sequencing

0. spike (throwaway, not committed): drive iroh under trio-asyncio/tractor.to_asyncio, echo bytes over a bi-stream between two procs. Fills in §1.1. Timebox it.
1. prep PR: annotation widening + rebind_from_sockname gate + transport_from_stream() tpt_key dispatch + ep= kwarg on start_listener() + lazy default_lo_addrs(). No new backend. Full suite green on tcp and uds.
2. _uniffi_trio.py + its tests (drive one iroh async call under bare trio.run(); assert no asyncio loop; assert cancellation frees the future).
3. QuicMsgStream + tests against a loopback iroh endpoint pair in one process (no tractor runtime): send/recv, clean EOF → b'', reset → BrokenResourceError, use-after-close → ClosedResourceError.
4. QuicListener + start_listener() + IrohAddress + key-file mgmt.
5. MsgpackQuicStream(MsgpackTransport) + connect_to() + maybe_open_context() connection pooling.
6. registration tables + --tpt-proto quic + full suite.
7. maddr + docs + a two-host example (pairs with #482’s format).

7. Testing

• capability predicate is_quic_available() → iroh importable and the uniffi driver symbol present at the pinned version. Same pytest.fail-early hook as plan 01 §7.2.
• the acceptance bar is the same: whole suite green under --tpt-proto quic. Expect this to shake out real bugs in the adapters (esp. teardown ordering and TransportClosed classification) — that’s the point.
• expect to need timeout headroom: iroh endpoint bind + first connect (relay discovery) is orders of magnitude slower than a UDS bind. Before touching any test deadline, rule out the CPU-throttle false-positive (see the project’s env_cpu_throttle_masquerades_as_regression note); then, if real, add a per-proto timeout multiplier to the test harness rather than editing individual tests.
• a no-network test mode: iroh with relays disabled + loopback direct addrs only, so CI doesn’t depend on n0’s infra. Make this the default in CI; mark the relay tests pytest.mark.net and keep them out of the default run.
• leak checks: assert every SecretKey/Endpoint is closed on actor teardown (an Endpoint left open holds UDP sockets and relay connections; a leak here shows up as hung tests, not errors).

8. Risks

risk	mitigation
uniffi codegen internals shift on upgrade	pinned minor, symbol assertion at import, the “no asyncio loop” test, documented fallback to to_asyncio
rust-thread callback → trio wakeup mishandled (segfault / lost wakeup / un-cancellable task)	strong ref on the ctypes trampoline; run_sync_soon only; bounded shielded cancel-drain; run the conc-anal skill over the bridge
iroh wheel availability for 3.13/3.14 on linux+macos	verify in step 0; if missing, that alone may force the aioquic fallback
QUIC latency/jitter destabilizes the existing suite’s timing assumptions	per-proto timeout multiplier, relay-less CI mode
(str, str) unwrapped form collides with UDS in wrap_address()	guarded case ordered first + explicit regression test (§3.2)
scope creep into iroh’s docs/blobs/gossip crates	this backend is Endpoint+Connection+bi-streams only; anything else is a separate issue

9. Follow-up issue seeds

• tractor.discovery delegating to iroh discovery (DNS/pkarr/mdns)
• per-Context QUIC sub-streams: today one Channel == one stream; QUIC would let each tractor.Context own its own stream with independent flow-control and cancellation — this is the genuinely novel win #353 gestures at, and it’s a runtime-layer change, not a transport one
• unreliable QUIC datagrams for a lossy-ok broadcast transport (pairs with plan 01’s TIPC-multicast seed)
• node-id allowlist → a real tractor authz story
• aioquic sans-io backend reusing §3’s adapters