# Plan 02 — QUIC backend via `iroh` FFI, uniffi-async rewritten onto `trio` Tracks gh [#353]. Prereq reading: [`00_shared_backend_contract.md`](./00_shared_backend_contract.md). **External-fact rule**: every claim here about `iroh`, UniFFI, generated bindings, QUIC wire/security behavior, or multiaddr support is provisional until the step-0 API-truth pass records a source or probe. Tractor/Trio behavior read from this checkout is the only locally proven basis for the plan. **Thesis**: the value of `iroh` over "just QUIC" is `NodeId`-addressed, NAT-traversing, relay-fallback endpoints — i.e. a `tractor` actor tree that spans hosts *without* a reachable listening socket. The cost is that `iroh`'s python surface is `uniffi`-generated **asyncio** and its listener is not a socket. This plan spends its complexity budget in exactly two places: a `trio`-native uniffi future bridge, and a `trio.abc.Listener`/`Stream` adapter pair. Everything else is contract boilerplate. [#353]: https://github.com/goodboy/tractor/issues/353 --- ## 1. Library selection (decided, with the rejected alternatives) **Chosen: `iroh` (PyPI, from `n0-computer/iroh-ffi`), pinned to a single minor.** The `iroh` python package is a `uniffi` binding over the rust `iroh` crate (QUIC via `quinn`/`noq`). Rejected, and why — record these so the next implementer doesn't relitigate: - **`aioquic`** (sans-io + asyncio): genuinely trio-portable (`hypercorn` already pairs its sans-io core with a trio UDP server, see the links in #353) and dependency-light. But it gives us *only* QUIC — no NodeId identity, no hole punching, no relay. We'd be reimplementing iroh's whole reason for existing. **Keep as the documented fallback** if the FFI bridge (§2) proves unmaintainable; the `MsgTransport` and `Listener` adapters from §3 are ~90% reusable against an `aioquic` core, which is a deliberate design property of this plan. - **`quiche` / `quinn` via a hand-rolled PyO3 ext**: strictly more work than reusing `iroh-ffi`, and puts us in the build-wheels business. - **`trio-asyncio`**: viable *shortcut* to run the asyncio-shaped bindings under trio, and `tractor` already ships infected-asyncio machinery (`tractor.to_asyncio`, `tests/test_infected_asyncio.py`). Rejected as the *primary* design because it makes every IPC send/recv cross a loop-boundary shim in the hot path, and because #353 asks explicitly for the asyncio support to be "rewritten for trio". **But**: build it first as the throwaway spike (§6 step 0) to de-risk the iroh API surface before writing the bridge. Version pinning: treat API stability across `iroh` minors as an **unverified external constraint** until step 0. Pin the version exercised by the spike to `iroh>=X.Y, str` | | | node addr (relay + direct) | `iroh.NodeAddr` | | | dial | `await endpoint.connect(node_addr, alpn)` | | | accept conn | `await endpoint.accept()` | | | open bi-stream | `await conn.open_bi()` | | | accept bi-stream | `await conn.accept_bi()` | | | send | `await send_stream.write_all(b)` | | | recv | `await recv_stream.read(n) -> bytes\|None` | | | half-close | `await send_stream.finish()` | | | endpoint close + completion | `close()` / `await closed()` | | | resolved node address | relay URL + direct socket addrs | | | future start/poll callback ABI | generated symbols + args | | | future cancel/complete/free | generated symbols + ordering | | | callback quiescence guarantee | after poll/complete/free? | | | cancellation terminal poll code | generated enum/value | | | iroh exception/status taxonomy | per operation | | --- ## 2. The `trio`-native uniffi future bridge (`tractor/ipc/_uniffi_trio.py`) ### 2.1 Step-0 generated-ABI gate The expected generated shape is: start an opaque Rust future, poll it with a C callback, cancel through a generated cancel symbol, consume its terminal value/status through `complete`, then call `free`. The expected callback may arrive on a foreign Rust thread. **All of that is external and provisional.** Step 0 must identify the exact generated driver and prove, from its template/source plus probes: 1. the start, poll, cancel, complete, and free signatures for every return-type family used by `iroh`; 2. poll result values and whether callbacks can be synchronous, concurrent, repeated, or late; 3. which terminal state permits `complete`, when `free` is legal, and when no callback can still reference Python; 4. whether generated callback-data and call-status objects must remain alive, and how generated lifting/errors are applied; 5. whether one narrow generated async-driver entrypoint can be replaced without importing or requiring an asyncio loop. Do not implement from a remembered UniFFI version. If cancel does not have a documented path to a terminal, safely freeable state, the native Trio bridge fails the spike gate and the first backend uses the infected-asyncio fallback. ### 2.2 Cancellation-safe ownership Do not let the caller task own a raw handle across an `await`. Introduce an actor-scoped `UniffiFutureSupervisor` running in the dedicated transport nursery specified in §3.2.1. That nursery must span parent bootstrap, the service nurseries, and final deregistration. For each call, its operation task owns the **entire** generated lifecycle: ```text create handle -> poll/callback loop -> complete -> lift/status -> free -> publish result ^ cancel request uses generated cancel, then follows the verified terminal poll/complete/free protocol ``` The operation task, not the awaiting caller, creates the handle. Creation and insertion in the supervisor's live-operation set must have no cancellation checkpoint between them. The operation retains strong references to the C callback trampoline, callback data, wake state, call status, and handle until step 0 proves all callbacks are quiescent and `free` has returned. Use one stable callback per operation unless the verified ABI requires a fresh one per poll; in either case, retain every potentially callable trampoline. Capture `current_trio_token()` in the Trio owner and schedule the wake into Trio with `token.run_sync_soon(...)`; the foreign callback only stores its poll result and schedules that wake. Caller cancellation is a request, not handle ownership transfer: 1. the caller sends an idempotent cancel request and waits under a short shield for the operation to acknowledge it; 2. the owner invokes the generated cancel function exactly once and continues the **verified** poll/complete/free sequence; 3. once caller cancellation is observed, cleanup completion never wins the race by returning a value. After acknowledgement the caller continues propagating its original Trio cancellation; if cleanup outlives the grace period it first abandons its result channel while the actor supervisor keeps ownership; 4. actor endpoint teardown stops accepting new calls, requests cancellation of all live operations, and joins the supervisor before destroying endpoint/key state. There is deliberately no `move_on_after(...): free(handle)` path. A timeout proves only that cleanup is slow; it does not prove that callbacks are quiescent or that `free` is legal. A wedged operation therefore remains visible in the supervisor and can delay graceful actor shutdown; process-level termination is the final escalation, not an unsafe FFI free. Structured-concurrency race to test: caller cancellation may land after handle creation, after each poll, during callback delivery, after terminal readiness, during `complete`, and before result publication. At every checkpoint exactly one operation task owns the handle, exactly one `free` is possible, and the supervisor cannot exit while that task or a callable trampoline remains. ### 2.3 how to apply it to the generated bindings Do **not** fork/vendor the generated `iroh` Python. Subject to the step-0 gate, ship a *narrow* re-dispatch shim: - write `tractor/ipc/_uniffi_trio.py` with the supervisor and a `@cm patch_uniffi_for_trio()` that patches only the generated async-driver entrypoint recorded in §1.1; - verify at import time that the expected symbol exists and raise a clear, actionable error naming the pinned `iroh` version if not. A silent fallback to asyncio would be a nightmare to debug. - treat every `iroh`/UniFFI upgrade as requiring the step-0 ABI gate again. Keep a test that drives one trivial call under bare `trio.run()`, asserts no asyncio loop, and injects cancellation at every lifecycle checkpoint. Point the module docstring at the exact generated template/revision mirrored by the shim. If step 0 reveals the generated code is *structurally* hostile to this (e.g. `asyncio` imported and used at module scope for more than the driver), fall back to option (b): run iroh under `tractor.to_asyncio` infected mode and open the follow-up to revisit. Say so in the PR rather than fighting it. --- ## 3. Mapping QUIC onto `MsgTransport` ### 3.1 the layering decision QUIC natively multiplexes streams inside one connection. The mapping that preserves *all* existing `tractor` semantics with the least new code: ``` iroh Endpoint == one per actor (process) -> the "listener" iroh Connection == one per peer actor -> pooled iroh bi-stream == one `Channel`/`MsgTransport` -> 1:1 ``` - keep the 4-byte ` None: ... async def send_all(self, data: bytes) -> None: ... async def wait_send_all_might_not_block(self) -> None: ... async def receive_some(self, max_bytes: int|None = None) -> bytes: ... async def send_eof(self) -> None: ... async def aclose(self) -> None: ... ``` Non-negotiable behaviours (each maps to a `match` case that already exists in `_transport.py` and must keep working): - `receive_some()` returns `b''` at clean EOF → `MsgpackTransport._iter_packets()` sees `header == b''` and raises `TransportClosed(loglevel='transport')`. **This is the graceful-disconnect path the whole runtime relies on**; get it right first. - a reset/aborted stream → raise `trio.BrokenResourceError`. - use after local close → raise `trio.ClosedResourceError` (ideally with `'another task closed this fd'`-equivalent text absent, so the `raise_on_report` branch at `_transport.py:290` stays quiet). - `send_all()` on a closed peer → `trio.BrokenResourceError`. - honour Trio's one-task-per-direction rule with public, implementation-local guards that raise `trio.BusyResourceError`; do not depend on `trio._util`. `MsgpackTransport` already serializes sends, while receives are single-task by construction. - **buffering**: if iroh's `read()` doesn't support "read up to n", `receive_some()` must maintain an internal leftover buffer. Note `MsgpackTransport` wraps us in `tricycle.BufferedReceiveStream` anyway, so `receive_some()` just needs *some* nonzero-progress contract. Centralize exception translation at every iroh/UniFFI boundary; no generated exception may escape into `Channel` or server code. Step 0 must record actual exception classes/status payloads and build an exhaustive operation-specific mapping: | observed condition | adapter result | | --- | --- | | receive clean EOF | `b''` | | local stream/listener/endpoint already closed | `trio.ClosedResourceError` | | concurrent same-direction operation | `trio.BusyResourceError` | | peer reset, stopped stream, lost connection | `trio.BrokenResourceError` | | dial rejected or no usable route | `ConnectionRefusedError` or `ConnectionError` | | caller's Trio deadline/cancellation | preserve Trio cancellation semantics | | unexpected FFI status/panic | chained `RuntimeError` identifying operation and pinned version | Preserve the original exception as `__cause__`, but sanitize messages so `_transport.py` sees stable Trio/Tractor categories, not version-specific iroh text. Endpoint accept failure becomes a listener `BrokenResourceError`; normal endpoint shutdown becomes `ClosedResourceError`. Add one test per observed step-0 status. ```python class QuicListener(trio.abc.Listener): ''' Accepts iroh `Connection`s and yields one `QuicMsgStream` per accepted bi-stream, so `trio.serve_listeners()` spawns one `handle_stream_from_peer()` per `Channel`. ''' async def accept(self) -> QuicMsgStream: ... async def aclose(self) -> None: ... ``` The accept-side subtlety is fan-out: one actor transport accepts connections and each connection accepts streams, while `Listener.accept()` returns one stream. Give **each** listener a supervisor task started with `await server_ep.listen_tn.start(...)`. That task creates and owns a cancel scope, a child nursery for the endpoint feeder plus per-connection feeders, a guarded stream queue, and a completion event. `start_listener(addr=, server_ep=, actor_tpt=)` does not return until the supervisor has reported all of those ready. Do not borrow an implicit parent nursery or spawn feeders lazily from `accept()`. The queue is a guarded `deque`, not an unowned memory-channel buffer. A feeder transfers a fully constructed, lease-owning stream into it only while the listener is open; if close wins the race, the feeder closes the stream itself. `accept()` atomically pops one item or waits on the queue condition. Once close is marked and the queue is empty, it raises `trio.ClosedResourceError`. `QuicListener.aclose()` is idempotent and has this exact order: 1. under the queue guard, mark closed and wake all `accept()` waiters without a checkpoint between the state change and notification; 2. cancel the listener-owned supervisor scope; 3. the supervisor's shielded `finally` joins the endpoint and all connection feeders, atomically detaches the queue, closes every queued stream, releases their leases, and closes the queue; 4. only after that finalizer finishes, the supervisor sets its completion event; 5. `aclose()` waits under a shield for that event and returns; concurrent closers wait for the same event. The same supervisor finalizer runs if its parent nursery is cancelled before someone calls `aclose()`. This makes the supervisor, not an arbitrarily cancelled caller, the sole final cleanup owner. Test cancellation at feeder accept, stream construction, queue transfer, `accept()` wakeup, and each close checkpoint; no feeder may outlive the listener and no queued lease may survive completion. This needs two explicit, typed references in the module-level listener call: `server_ep=` is the IPC server `Endpoint` that owns `listen_tn`, while `actor_tpt=` is the already-open `QuicActorEndpoint` whose iroh accept API supplies connections. Store `actor_tpt` on the server endpoint during actor transport bootstrap and pass both keyword-only arguments; socket backends ignore `actor_tpt`. `Endpoint.start_listener()` then stores the listener's already-resolved address instead of calling `getsockname()`. ### 3.4 `maddr` Expected multiaddr spellings for direct QUIC and relay routes are **step-0 verification items**, not assumptions: ``` /ip4//udp/

/quic-v1 # direct /ip4//udp/

/quic-v1/p2p/ # direct + identity /dns//tcp/443/tls/ws/p2p/ # relay-ish ``` - Do not emit NodeId alone in the first backend: without enabled discovery it would discard the route required by the complete `IrohAddress`. `mk_maddr()` must preserve NodeId, ALPN, relay URL, and all direct addresses, or return a canonical Tractor string form that does until a multiaddr grammar can round-trip every field. - Verify whether an iroh NodeId can losslessly map to `/p2p/`. If not, use a tractor-local `/iroh/` segment rather than pretending to be a libp2p peer-id. This needs upstream registration, on the same track as `wg`/`tipc` (gh #483). - this backend is the strongest argument for gh #443's **tunnelled/composed maddr** item: `/ip4/../udp/../quic-v1/..` *is* a composed stack. Cross-reference plan 03 §5 so the two grammars land compatibly. --- ## 4. Discovery integration - The registrar stores the complete `IrohAddress`, not only a NodeId. Registration is forbidden until endpoint address resolution has produced that descriptor. If route hints change later, dynamic re-registration is a follow-up; the spike uses the pre-registration snapshot. - Optional iroh discovery mechanisms and their names/capabilities are step-0 verification items and out of scope for the first backend. No NodeId-only reachability claim is made. - Relay configuration belongs to `QuicActorEndpoint` creation, not `start_listener()`, because dialing and listening reuse the same endpoint. The demo's relay choice and self-hosted option are selected only after step 0 verifies the pinned API. ## 5. Security note The transport-security and NodeId-authentication properties of the pinned iroh stack are **step-0 documentation-verification items**. Claim only the properties supported by that version's source and docs. Two design consequences remain: 1. an **allowlist hook** — an actor should be able to reject inbound connections from unknown node-ids *before* the `Aid` handshake. Natural home: a predicate kwarg on `start_listener()`, evaluated in `QuicListener`'s connection acceptor task. Sketch it; ship it in PR 1 if cheap (it is). 2. do **not** claim any security property for the other backends by association. `tcp`/`uds`/`tipc` remain unauthenticated; that's what plan 03 (wg) is for. ## 6. Commit sequencing 0. **spike (throwaway, not committed)**: drive iroh under `trio-asyncio`/`tractor.to_asyncio`, echo bytes over a bi-stream between two processes. Fill §1.1 with generated ABI, endpoint resolution, close/join, and error observations. Probe cancel at every generated lifecycle phase. Timebox it and use the fallback if any mandatory ownership fact stays unknown. 1. prep PR: tagged address migration, annotation widening, non-socket listener reconciliation, `tpt_key` dispatch, typed `server_ep=`/`actor_tpt=` listener inputs, and lazy default addresses. **No new backend.** Keep tcp and uds behavior unchanged. 2. bootstrap prep: pass `ChildTransportBootstrap` through every process-launch path and add the transport nursery around child parent-dial, service, deregistration, and teardown. Resolve the endpoint address before registration. Add no iroh-specific global state. 3. `_uniffi_trio.py` supervisor + lifecycle fault-injection tests: no asyncio loop, one owner/complete/free, callback retention, bounded caller handoff, and joined durable cleanup. 4. `QuicActorEndpoint` + provisioned registrar descriptor + loopback direct-address tests; prove one endpoint handles dial, listen, address lookup, and ordered teardown. 5. `QuicMsgStream` + exhaustive error-normalization and lease release tests against the loopback endpoint pair. 6. `QuicListener` supervisor + cancellation-at-every-checkpoint tests, including queued-stream draining and feeder joins. 7. `MsgpackQuicStream`, full-key connection pooling, registration tables, and `--tpt-proto quic`; then run the full suite. 8. routable maddr/string form + docs + a two-host example (pairs with #482's format). ## 7. Testing - capability predicate `is_quic_available()` → `iroh` importable *and* every step-0-recorded driver symbol present at the pinned version. Same `pytest.fail`-early hook as plan 01 §7.2. - **the acceptance bar is the same**: whole suite green under `--tpt-proto quic`. Expect this to shake out real bugs in the adapters (esp. teardown ordering and `TransportClosed` classification) — that's the point. - Measure endpoint bind and first-connect latency in step 0; do not assume a multiplier. Before changing a deadline, rule out the project's CPU-throttle false-positive, then prefer one per-proto harness multiplier over individual-test edits. - Use the step-0-verified relay-disable configuration with direct loopback addresses for default CI. Mark separately verified relay tests `pytest.mark.net` and keep them out of default CI. - leak checks: assert the actor has one key/endpoint, every FFI operation completed/freed once, all listener feeders joined, all queued streams closed, every connection lease released, and endpoint close completion observed before actor teardown. - address-ordering check: block registration until a descriptor with NodeId, ALPN, and at least one route is published; reject sentinel, NodeId-only, and post-registration mutation cases. ## 8. Risks | risk | mitigation | | --- | --- | | uniffi codegen internals shift on upgrade | pinned minor, symbol assertion at import, the "no asyncio loop" test, documented fallback to `to_asyncio` | | callback wakeup/lifetime semantics differ from the hypothesis | step-0 source + probe gate; retain callback/data through verified quiescence; durable owner; never timeout-free | | cancelled foreign future never reaches a freeable state | bounded caller handoff to visible actor supervisor; joined graceful shutdown or process-level escalation; never speculative free | | `iroh` wheel availability for 3.13/3.14 on linux+macos | verify in step 0; if missing, that alone may force the `aioquic` fallback | | endpoint or route resolution is not ready before parent dial/registration | actor endpoint bootstrap barrier; publish only a complete resolved descriptor | | connection closes while a stream still uses it | full-key pool + stream-held leases + exact-once release tests | | listener close strands feeder tasks or queued streams | listener-owned scope/completion event; cancel, join, drain, then return | | QUIC latency/jitter destabilizes suite timing assumptions | measure first; per-proto multiplier only if demonstrated; relay-less CI mode | | address tuple collides with another backend | required `'quic'` tag and exact-shape dispatch | | scope creep into iroh's docs/blobs/gossip crates | this backend is `Endpoint`+`Connection`+bi-streams only; anything else is a separate issue | ## 9. Follow-up issue seeds - `tractor.discovery` delegating to iroh discovery (DNS/pkarr/mdns) - per-`Context` QUIC sub-streams: today one `Channel` == one stream; QUIC would let each `tractor.Context` own its own stream with independent flow-control and cancellation — this is the genuinely novel win #353 gestures at, and it's a runtime-layer change, not a transport one - unreliable QUIC datagrams for a lossy-ok broadcast transport (pairs with plan 01's TIPC-multicast seed) - node-id allowlist → a real `tractor` authz story - `aioquic` sans-io backend reusing §3's adapters