# Plan 02 — QUIC backend via `iroh` FFI, uniffi-async rewritten onto `trio` Tracks gh [#353]. Prereq reading: [`00_shared_backend_contract.md`](./00_shared_backend_contract.md). **Thesis**: the value of `iroh` over "just QUIC" is `NodeId`-addressed, NAT-traversing, relay-fallback endpoints — i.e. a `tractor` actor tree that spans hosts *without* a reachable listening socket. The cost is that `iroh`'s python surface is `uniffi`-generated **asyncio** and its listener is not a socket. This plan spends its complexity budget in exactly two places: a `trio`-native uniffi future bridge, and a `trio.abc.Listener`/`Stream` adapter pair. Everything else is contract boilerplate. [#353]: https://github.com/goodboy/tractor/issues/353 --- ## 1. Library selection (decided, with the rejected alternatives) **Chosen: `iroh` (PyPI, from `n0-computer/iroh-ffi`), pinned to a single minor.** The `iroh` python package is a `uniffi` binding over the rust `iroh` crate (QUIC via `quinn`/`noq`). Rejected, and why — record these so the next implementer doesn't relitigate: - **`aioquic`** (sans-io + asyncio): genuinely trio-portable (`hypercorn` already pairs its sans-io core with a trio UDP server, see the links in #353) and dependency-light. But it gives us *only* QUIC — no NodeId identity, no hole punching, no relay. We'd be reimplementing iroh's whole reason for existing. **Keep as the documented fallback** if the FFI bridge (§2) proves unmaintainable; the `MsgTransport` and `Listener` adapters from §3 are ~90% reusable against an `aioquic` core, which is a deliberate design property of this plan. - **`quiche` / `quinn` via a hand-rolled PyO3 ext**: strictly more work than reusing `iroh-ffi`, and puts us in the build-wheels business. - **`trio-asyncio`**: viable *shortcut* to run the asyncio-shaped bindings under trio, and `tractor` already ships infected-asyncio machinery (`tractor.to_asyncio`, `tests/test_infected_asyncio.py`). Rejected as the *primary* design because it makes every IPC send/recv cross a loop-boundary shim in the hot path, and because #353 asks explicitly for the asyncio support to be "rewritten for trio". **But**: build it first as the throwaway spike (§6 step 0) to de-risk the iroh API surface before writing the bridge. Version pinning: `iroh` moves fast and has had breaking API renames across minors. Pin `iroh>=X.Y, str` | | | node addr (relay + direct) | `iroh.NodeAddr` | | | dial | `await endpoint.connect(node_addr, alpn)` | | | accept conn | `await endpoint.accept()` | | | open bi-stream | `await conn.open_bi()` | | | accept bi-stream | `await conn.accept_bi()` | | | send | `await send_stream.write_all(b)` | | | recv | `await recv_stream.read(n) -> bytes\|None` | | | half-close | `await send_stream.finish()` | | --- ## 2. The `trio`-native uniffi future bridge (`tractor/ipc/_uniffi_trio.py`) ### 2.1 what uniffi actually generates `uniffi`'s async support does not use asyncio *semantically* — it uses asyncio only as the *executor* for a poll loop. The generated python for an `async fn` is, in shape: 1. call `_uniffi_..._(...)` → returns an opaque `RustFuture` handle (a `void*`/`u64`). 2. loop: call `ffi_..._rust_future_poll_(handle, callback, callback_data)`. The callback is a C-ABI fn pointer invoked **from an arbitrary rust thread** with a poll-result code (`READY`/`MAYBE_READY`). 3. the generated glue's callback resolves an `asyncio.Future` via `loop.call_soon_threadsafe(...)`; the coroutine awaits it, then re-polls. 4. on ready: `ffi_..._rust_future_complete_(handle, &call_status)` → the value; then `ffi_..._rust_future_free_(handle)`. **The asyncio dependency is confined to step 3.** That is the whole insight: the bridge is ~40 lines. ### 2.2 the trio version ```python async def await_rust_future( poll: Callable, # ffi_..._rust_future_poll_ complete: Callable, # ffi_..._rust_future_complete_ free: Callable, # ffi_..._rust_future_free_ handle: int, lift: Callable[[Any], Any], ) -> Any: ''' Drive a `uniffi` rust-future to completion on the current `trio` task, bridging rust-thread wakeups via `TrioToken.run_sync_soon()`. ''' token = trio.lowlevel.current_trio_token() while True: wake = trio.Event() # NOTE, invoked from a *rust* thread! def _cb(_data, poll_code): token.run_sync_soon(wake.set) cb = _UNIFFI_FUTURE_CALLBACK(_cb) # keep a strong ref! poll(handle, cb, 0) await wake.wait() if : break try: status = _UniffiRustCallStatus.default() res = complete(handle, status) _uniffi_check_call_status(status) # reuse generated helper return lift(res) finally: free(handle) ``` Critical details, each a real bug if missed: - **`token.run_sync_soon()` is the only trio API callable from a foreign thread**, and it is documented as such. Use it; do *not* use `trio.from_thread.run_sync` (requires a trio thread context) and do not touch the `Event` directly from the callback. - **the poll code must reach the trio side.** Capture it in a `nonlocal`/1-slot list written by the callback *before* `run_sync_soon`, since the callback owns the value. Handle `MAYBE_READY` by re-polling (the loop above does). - **keep the `ctypes` callback object alive** across the await — a GC'd `CFUNCTYPE` trampoline is a segfault. Bind it to a local *and* make sure the local outlives the `poll()` call window. - **cancellation.** `await wake.wait()` is a trio checkpoint, so a `Cancelled` can fire while rust still owns the future. On cancel we must still `free(handle)` — and per uniffi, the correct sequence is to call the generated `ffi_..._rust_future_cancel_(handle)` then continue polling to completion before `free`. Wrap the whole thing so the cancel path does: `with trio.CancelScope(shield=True): cancel(handle); ; free(handle)`. **Bounded** shield (add a `trio.move_on_after()` with a module-level constant) so a wedged rust future can't make an actor un-cancellable — `tractor` is SC-first and an unbounded shield here would violate that. - **`trio.lowlevel.current_trio_token()`** must be captured on the trio side (not in the callback). ### 2.3 how to apply it to the generated bindings Do **not** fork/vendor the generated `iroh` python. Instead ship a *narrow* re-dispatch shim: - write `tractor/ipc/_uniffi_trio.py` with `await_rust_future()` plus a `@cm patch_uniffi_for_trio()` that monkey-patches the generated module's single async-driver entrypoint (in current uniffi that's `_uniffi_rust_call_async` / `_rust_call_async`, one function) to the trio implementation. - verify at import time that the expected symbol exists and raise a clear, actionable error naming the pinned `iroh` version if not. A silent fallback to asyncio would be a nightmare to debug. - **plan for this to break on `iroh`/`uniffi` upgrades.** Mitigate with (a) a unit test that drives one trivial `iroh` async call under bare `trio.run()` and asserts no event loop was ever created (`asyncio.get_event_loop_policy()` untouched / `asyncio._get_running_loop() is None`), and (b) a docstring pointing at the uniffi codegen template this mirrors. If step 0 reveals the generated code is *structurally* hostile to this (e.g. `asyncio` imported and used at module scope for more than the driver), fall back to option (b): run iroh under `tractor.to_asyncio` infected mode and open the follow-up to revisit. Say so in the PR rather than fighting it. --- ## 3. Mapping QUIC onto `MsgTransport` ### 3.1 the layering decision QUIC natively multiplexes streams inside one connection. The mapping that preserves *all* existing `tractor` semantics with the least new code: ``` iroh Endpoint == one per actor (process) -> the "listener" iroh Connection == one per peer actor -> pooled iroh bi-stream == one `Channel`/`MsgTransport` -> 1:1 ``` - keep the 4-byte ` (node_id_str, alpn_str)`** — a `(str, str)` tuple, which is *unambiguously distinct* from `TCPAddress`'s `(str, int)`. But careful: `wrap_address()`'s UDS case is `case (_, filename) if type(filename) is str` — which **already catches `(str, str)`**. So the iroh `case` MUST be ordered *before* the UDS case and guarded, e.g. `case (str() as nid, str() as alpn) if _is_node_id(nid):` with `_is_node_id()` a cheap length+alphabet check. Add a regression test asserting a UDS `(dir, filename)` pair still wraps to `UDSAddress` — this is the exact "wrong transport loaded" hazard `_addr.py:214` warns about. - `.bindspace` → `self._alpn`. This is the honest analogue: the ALPN is the set of endpoints willing to talk to you, and two `tractor` deployments sharing an iroh network are separated by ALPN exactly as two UDS deployments are separated by directory. Include a `tractor` version/proto epoch in the default ALPN so incompatible runtimes can't handshake. - `.is_valid` → node-id parses, alpn non-empty. - **`get_root()` is the hard one.** There is no well-known-port analogue: an iroh node id is a *keypair*, so "the host's default registrar addr" requires a *persisted secret key*. Design: - the root/registrar's secret key lives at `get_rt_dir() / 'iroh_registrar.key'` (0600), created on first use. - `get_root()` must stay **pure and import-time-safe** (contract §2.3: `_default_lo_addrs` is built at import!). So `get_root()` *reads* the key file if present and otherwise returns an `IrohAddress` with `_node_id=''`/sentinel, and the **generation** happens in an explicit sibling — `ensure_registrar_key() -> IrohAddress` — called from the listen path. Pure getter, explicit setter; do not smuggle key generation into `get_root()`. - this almost certainly means `_default_lo_addrs` must become lazy for this backend. **Land that refactor as its own prep commit** (a `default_lo_addrs()` that computes per-call instead of the import-time dict) — it also unblocks plan 03's netns-scoped defaults. - `get_random()`: generate a fresh `SecretKey` per subactor and return its node-id. Note this runs post-fork pre-listen (contract §4) and costs an ed25519 keygen (~µs, fine). The *secret* can't live in a frozen `Address`, so it must be stashed where the listen path can find it: a module-level `dict[node_id, SecretKey]` populated by `get_random()` and consumed+popped by `start_listener()`. Ugly but honest; document it and note the alternative (thread the key through `Endpoint`) as a follow-up. #### 3.2.1 `proto_key`: `'iroh'` vs `'quic'` Use **`'quic'`** for the `proto_key`/`--tpt-proto` name and name the module `_quic.py`, with `iroh` as the *implementation*. Rationale: it keeps the door open for the `aioquic` fallback (§1) without a user-visible rename, and it matches how `uds` is a proto name rather than a lib name. Put `iroh`-specific bits behind an internal `_iroh` submodule if the file gets big. ### 3.3 the `trio.abc` adapters — where the real work is Contract §3 says a non-socket backend needs three upstream generalizations. Land them **as a prep PR, before any iroh code**, so they can be reviewed on their own merits with tcp/uds still the only backends: 1. **`Endpoint.start_listener()` must not assume `.socket.getsockname()`.** Use the same `Address.rebind_from_sockname: ClassVar[bool]` gate that plan 01 §3.2 introduces — coordinate so it lands once. (If plan 01 lands first, this is free.) 2. **`transport_from_stream()` (`_types.py:92`) must not assume `trio.SocketStream`.** Replace the `sock.family` match with: check `isinstance(stream, trio.SocketStream)` → existing family match; else look for a `stream.tpt_key: ClassVar[MsgTransportKey]` attribute on the adapter and use it. Keeps the existing path byte-identical and makes new stream types self-describing (a much better shape than growing an `isinstance` ladder). 3. **type annotations**: `handle_stream_from_peer(stream: trio.SocketStream)` → `trio.abc.Stream`; `Endpoint._listener: SocketListener|None` → `trio.abc.Listener|None`; `MsgTransport.stream: trio.SocketStream` → `trio.abc.Stream`. Annotation-only, zero behaviour change. Then the adapters: ```python class QuicMsgStream(trio.abc.HalfCloseableStream): ''' A single `iroh` bi-directional QUIC stream presented as a `trio` byte-stream so `MsgpackTransport` can frame over it unmodified. ''' tpt_key: ClassVar[MsgTransportKey] = ('msgpack', 'quic') def __init__(self, conn, send, recv) -> None: ... async def send_all(self, data: bytes) -> None: ... async def wait_send_all_might_not_block(self) -> None: ... async def receive_some(self, max_bytes: int|None = None) -> bytes: ... async def send_eof(self) -> None: ... async def aclose(self) -> None: ... ``` Non-negotiable behaviours (each maps to a `match` case that already exists in `_transport.py` and must keep working): - `receive_some()` returns `b''` at clean EOF → `MsgpackTransport._iter_packets()` sees `header == b''` and raises `TransportClosed(loglevel='transport')`. **This is the graceful-disconnect path the whole runtime relies on**; get it right first. - a reset/aborted stream → raise `trio.BrokenResourceError`. - use after local close → raise `trio.ClosedResourceError` (ideally with `'another task closed this fd'`-equivalent text absent, so the `raise_on_report` branch at `_transport.py:290` stays quiet). - `send_all()` on a closed peer → `trio.BrokenResourceError`. - honour `trio`'s one-task-per-direction rule: guard with `trio._util.ConflictDetector` equivalents (or just document + assert), because `MsgpackTransport` already serializes sends with a `StrictFIFOLock` but recvs are single-task by construction. - **buffering**: if iroh's `read()` doesn't support "read up to n", `receive_some()` must maintain an internal leftover buffer. Note `MsgpackTransport` wraps us in `tricycle.BufferedReceiveStream` anyway, so `receive_some()` just needs *some* nonzero-progress contract. ```python class QuicListener(trio.abc.Listener): ''' Accepts iroh `Connection`s and yields one `QuicMsgStream` per accepted bi-stream, so `trio.serve_listeners()` spawns one `handle_stream_from_peer()` per `Channel`. ''' async def accept(self) -> QuicMsgStream: ... async def aclose(self) -> None: ... ``` The accept-side subtlety: `trio.abc.Listener.accept()` yields one stream per call, but iroh gives us *connections* which then yield *streams*. So `QuicListener` needs an internal `trio.MemoryReceiveChannel[QuicMsgStream]` fed by a background task-pair (one task accepting connections, one per connection accepting bi-streams). `trio.abc.Listener` has no nursery, so: make the listener **constructed by an `@acm`** that owns the nursery, and have `start_listener()` be that `@acm`'s driver. ⚠️ this collides with `Endpoint.start_listener()` being a plain `async def` returning a listener. Two options: - **(a)** hang the nursery off the `Endpoint`'s existing `listen_tn` — `_serve_ipc_eps()` already creates `listen_tn` and passes it into every `Endpoint` (`_server.py:1063-1074`), and `Endpoint.listen_tn` is right there. So `start_listener()` can `self.listen_tn.start_soon(...)` the acceptor tasks. **Recommended**: no upstream signature change, correct lifetime (dies with the ep group), and it's why `listen_tn` is on the struct in the first place. - (b) change `start_listener()` to a `@acm`. Bigger blast radius; only if (a) proves insufficient. Since `start_listener()` is called via `inspect.getmodule(addr)` with only `addr=` (contract §1.3), option (a) needs the `Endpoint` itself. Either add `ep=` to the module-level `start_listener()` call signature (all backends ignore it except quic → small upstream change, do it as part of the prep PR and make it keyword-only with a default) or have `QuicListener.accept()` lazily spawn via `trio.lowlevel.current_task().parent_nursery` (**rejected** — fragile, implicit). Do the explicit `ep=` kwarg. ### 3.4 `maddr` Multiaddr already standardizes the pieces: ``` /ip4//udp/

/quic-v1 # direct /ip4//udp/

/quic-v1/p2p/ # direct + identity /dns//tcp/443/tls/ws/p2p/ # relay-ish ``` - primary form: `/p2p/` alone is a legal maddr and is the *only* required component for iroh dialling — relay + direct addrs are discovery hints. So `mk_maddr()` emits `/p2p/` and, when known, prefixes the direct `/ip4/../udp/../quic-v1/`. - `/p2p/` values are multihash-encoded peer ids; an iroh node-id is a raw ed25519 key. Converting requires the identity multihash + libp2p key protobuf wrapper. **Decide**: emit the raw node-id under a *tractor-local* `/iroh/` segment (needs upstream registration, same track as `wg`/`tipc`, gh #483) rather than pretending to be a libp2p peer-id we can't round-trip. Return the `str` form until upstream lands (`MsgTransport.maddr` is `Multiaddr|str`). - this backend is the strongest argument for gh #443's **tunnelled/composed maddr** item: `/ip4/../udp/../quic-v1/..` *is* a composed stack. Cross-reference plan 03 §5 so the two grammars land compatibly. --- ## 4. Discovery integration - iroh's node-id addressing means the `tractor` registrar can hold `IrohAddress`es that are **reachable from anywhere** with no port-forwarding — that is the headline feature. The registrar itself works unchanged. - iroh has its own discovery (DNS/pkarr/mdns). **Out of scope**; note in the follow-up that `tractor.discovery` could eventually delegate to it, which would be the direct analogue of plan 01's TIPC-topology idea. - relay servers: default to n0's public relays for the demo, document self-hosting (docs.iroh.computer's dedicated-infra page is linked from #353), and make the relay set a `start_listener()` kwarg. ## 5. Security note QUIC is TLS-1.3-always and iroh authenticates by node-id, so this backend is the first `tractor` transport with real transport security and peer authentication. Two things follow: 1. an **allowlist hook** — an actor should be able to reject inbound connections from unknown node-ids *before* the `Aid` handshake. Natural home: a predicate kwarg on `start_listener()`, evaluated in `QuicListener`'s connection acceptor task. Sketch it; ship it in PR 1 if cheap (it is). 2. do **not** claim any security property for the other backends by association. `tcp`/`uds`/`tipc` remain unauthenticated; that's what plan 03 (wg) is for. ## 6. Commit sequencing 0. **spike (throwaway, not committed)**: drive iroh under `trio-asyncio`/`tractor.to_asyncio`, echo bytes over a bi-stream between two procs. Fills in §1.1. Timebox it. 1. prep PR: annotation widening + `rebind_from_sockname` gate + `transport_from_stream()` `tpt_key` dispatch + `ep=` kwarg on `start_listener()` + lazy `default_lo_addrs()`. **No new backend.** Full suite green on tcp *and* uds. 2. `_uniffi_trio.py` + its tests (drive one iroh async call under bare `trio.run()`; assert no asyncio loop; assert cancellation frees the future). 3. `QuicMsgStream` + tests against a *loopback* iroh endpoint pair in one process (no `tractor` runtime): send/recv, clean EOF → `b''`, reset → `BrokenResourceError`, use-after-close → `ClosedResourceError`. 4. `QuicListener` + `start_listener()` + `IrohAddress` + key-file mgmt. 5. `MsgpackQuicStream(MsgpackTransport)` + `connect_to()` + `maybe_open_context()` connection pooling. 6. registration tables + `--tpt-proto quic` + full suite. 7. maddr + docs + a two-host example (pairs with #482's format). ## 7. Testing - capability predicate `is_quic_available()` → `iroh` importable *and* the uniffi driver symbol present at the pinned version. Same `pytest.fail`-early hook as plan 01 §7.2. - **the acceptance bar is the same**: whole suite green under `--tpt-proto quic`. Expect this to shake out real bugs in the adapters (esp. teardown ordering and `TransportClosed` classification) — that's the point. - expect to need **timeout headroom**: iroh endpoint bind + first connect (relay discovery) is orders of magnitude slower than a UDS bind. Before touching any test deadline, rule out the CPU-throttle false-positive (see the project's `env_cpu_throttle_masquerades_as_regression` note); then, if real, add a per-proto timeout multiplier to the test harness rather than editing individual tests. - a no-network test mode: iroh with relays disabled + loopback direct addrs only, so CI doesn't depend on n0's infra. **Make this the default in CI**; mark the relay tests `pytest.mark.net` and keep them out of the default run. - leak checks: assert every `SecretKey`/`Endpoint` is closed on actor teardown (an `Endpoint` left open holds UDP sockets and relay connections; a leak here shows up as hung tests, not errors). ## 8. Risks | risk | mitigation | | --- | --- | | uniffi codegen internals shift on upgrade | pinned minor, symbol assertion at import, the "no asyncio loop" test, documented fallback to `to_asyncio` | | rust-thread callback → trio wakeup mishandled (segfault / lost wakeup / un-cancellable task) | strong ref on the ctypes trampoline; `run_sync_soon` only; **bounded** shielded cancel-drain; run the `conc-anal` skill over the bridge | | `iroh` wheel availability for 3.13/3.14 on linux+macos | verify in step 0; if missing, that alone may force the `aioquic` fallback | | QUIC latency/jitter destabilizes the existing suite's timing assumptions | per-proto timeout multiplier, relay-less CI mode | | `(str, str)` unwrapped form collides with UDS in `wrap_address()` | guarded case ordered first + explicit regression test (§3.2) | | scope creep into iroh's docs/blobs/gossip crates | this backend is `Endpoint`+`Connection`+bi-streams only; anything else is a separate issue | ## 9. Follow-up issue seeds - `tractor.discovery` delegating to iroh discovery (DNS/pkarr/mdns) - per-`Context` QUIC sub-streams: today one `Channel` == one stream; QUIC would let each `tractor.Context` own its own stream with independent flow-control and cancellation — this is the genuinely novel win #353 gestures at, and it's a runtime-layer change, not a transport one - unreliable QUIC datagrams for a lossy-ok broadcast transport (pairs with plan 01's TIPC-multicast seed) - node-id allowlist → a real `tractor` authz story - `aioquic` sans-io backend reusing §3's adapters