Document TIPC-over-`wg`, add a handoff doc

Anticipating gh #502 — TIPC over a WireGuard mesh as our go-to
multihost tpt deployment — plus a cold-start handoff for whoever
(or whatever) picks this up next.

The wg deats, both verified locally,
- a wg iface is L3/`tun` (`POINTOPOINT,NOARP`, `link/none`, no
  L2 addr) so TIPC's `eth` media **cannot** bind it; the udp
  bearer is *mandatory* over wg, not merely an alternative.
  Also its ~1420 MTU sits under ethernet's 1500.
- the composed deployment maddr is
  `/ip4/<pub>/udp/51820/wg/u<key>/tipc/<stype>/<inst>/<scope>`.
  XXX note the tipc segment has NO locative part unlike tcp's
  inner `/ip4/../tcp/..` — a service name is
  location-independent, so wg carries routing and tipc carries
  identity. That's the argument for one `/tipc` proto w/ a
  structured value in the #498 spec proposal.

XXX ALSO correcting a premise: TIPC is **not** unencrypted. It
ships AES-GCM crypto (`tipc node set key`, linux 5.9+) w/
cluster/master/per-node keys + rekeying. Those keys are
symmetric+pre-shared tho, so wg is still preferred for
public-key identity, NAT traversal, and one overlay every tpt
can share.

`01_tipc_HANDOFF.md` is deliberately provider-neutral: env
setup, the hard-won kernel facts table, the two closed design
decisions (+why), what landed, the pre-land TODOs and the repo's
working conventions.

(this patch was generated in some part by `claude-code` using `claude-opus-5` (`anthropic`))
wkt/pr493_review
Gud Boi 2026-08-17 17:38:05 -04:00
parent 1802641e41
commit 4aa7a890cb
4 changed files with 326 additions and 6 deletions

View File

@ -0,0 +1,249 @@
# TIPC backend — handoff
Status: **PR [#493] is feature-complete and green**; what remains
is landing logistics plus a named follow-up track.
Audience: any agent or human picking this up cold, from any
provider. Nothing here assumes a particular harness or tooling.
Read in this order,
1. this file (orientation + what's already settled)
2. [`00_shared_backend_contract.md`](./00_shared_backend_contract.md)
**normative** description of what a `tractor` transport
backend *is*
3. [`01_tipc_backend.md`](./01_tipc_backend.md) — the plan,
already reconciled against the live-kernel findings
Do **not** re-derive the design or re-select libraries. Where
this doc and the code disagree, **the code wins** — fix the doc
in the same change (contract §0).
[#493]: https://github.com/goodboy/tractor/pull/493
---
## 1. Environment
The backend needs a linux kernel module that is **not loaded by
default**:
```bash
sudo modprobe tipc
tipc node get address # confirms the module is live
```
Everything else is stdlib — TIPC adds **zero** dependencies.
This repo is a git worktree with a `uv`-managed venv at
`./py313`. Run things through it:
```bash
./py313/bin/python -m pytest tests/ipc -q
./py313/bin/python -m pytest tests/ -q --tpt-proto tipc
```
To rebuild docs you need the docs dep-group, which *mutates*
that venv — `uv sync` afterwards to restore it:
```bash
UV_PROJECT_ENVIRONMENT=py313 uv run --group docs \
python -m sphinx -b html docs /tmp/docbuild
UV_PROJECT_ENVIRONMENT=py313 uv sync
```
Without the module, `--tpt-proto tipc` fails loudly and
immediately (by design); `tests/ipc/test_tipc.py`'s
kernel-touching cases self-skip.
## 2. What the backend is, in three sentences
An actor's TIPC address is a **service name** `(stype, instance)`
— no host, no port. Binding the singleton `TIPC_ADDR_NAMESEQ`
range *publishes* it into a kernel-maintained cluster-wide name
table (visible via `tipc nametable show`), and a peer's
`.connect()`-by-name *is* the discovery lookup, resolved
in-kernel. `TIPC_ADDR_ID` port-ids are only ever **observed**,
never user-facing.
Everything lives in `tractor/ipc/_tipc.py`.
## 3. Hard-won facts — do not re-litigate these
All verified against a live kernel. Several contradict what the
plan originally assumed.
| fact | why it matters |
| --- | --- |
| A duplicate name bind **succeeds**, and connects **round-robin** between publishers | An instance collision is *silent crosstalk*, never `EADDRINUSE`. Hence the `blake2b` instance digest. |
| Dialing an unpublished name gives `EHOSTUNREACH` **instantly**, as a **bare `OSError`** — not a `ConnectionError` subtype | `_reraise_as_connerr()` is REQUIRED by contract §4, not polish |
| `getsockname()` always answers a `TIPC_ADDR_ID` port-id, even pre-bind | why `TIPCAddress.rebind_from_sockname = False` |
| A connect-then-drop peer makes `getpeername()` raise `ENOTCONN` | Unguarded, this **kills the whole actor**`.get_stream_addrs()` runs *before* the handshake, so it escapes handshake tolerance. See `_maybe_sockaddr()`. |
| `SO_ACCEPTCONN` works fine (answers `1`) | trio's `except OSError` carve-out is not load-bearing here |
| Graceful peer close arrives as `BrokenResourceError`/`ECONNRESET`, not a clean 0-byte EOF | benign; `_iter_packets()` already classifies it as a normal disconnect |
| Topology `struct tipc_event` is **48 bytes** (`4+4+4+8+28`) | the plan said 40 |
| The topology server **accepts native `'='` byte-order** | the plan's proposed `'>'`-retry endianness probe was deleted as unnecessary |
| `TIPC_WAIT_FOREVER` is `-1` in python | must be masked (`& 0xFFFFFFFF`) before packing as unsigned |
| TIPC **has** AES-GCM encryption (`tipc node set key`, linux 5.9+) | cluster/master/per-node keys + rekeying. Symmetric pre-shared, which is why a wg mesh is still preferred — see §6. |
| A wg interface is L3/`tun` (`POINTOPOINT,NOARP`, `link/none`) | TIPC's `eth` media **cannot** bind it; the udp bearer is mandatory over wg |
Two design decisions that are **closed**, with reasons:
- **The accepting side does not learn the peer's service name.**
A port-id can't be reversed into one. It gets a
`TIPC_NAME_UNKNOWN` sentinel plus the observed `(node, ref)`,
and that's fine — the `Aid` from the handshake already carries
the peer's logical identity. (`uds` has the same wart.)
- **Do not fold digest bits into the service `_stype` to widen
the collision space.** A topology subscription can only watch
**one** `stype`, so varying it per-actor would need 65536
subscriptions and kills the push-registry work outright. If
crosstalk ever bites, the escalation is a post-bind
verification handshake ([#501]).
[#501]: https://github.com/goodboy/tractor/issues/501
## 4. What landed
15 commits, `22ef362d..2d082373`, on `wkt/tipc_backend_378`,
based on `ng_tpts_planning` (PR [#492], docs-only).
- `TIPCAddress` + `is_tipc_available()` + `start_listener()`
- `MsgpackTIPCStream` (`connect_to()`, `get_stream_addrs()`)
- `open_topology_events()` — the `TIPC_TOP_SRV` push feed
- registration across every table in contract §2
- interim `str`-only `/tipc/…` maddr grammar
- a `--tpt-proto=tipc` CI leg, **non-blocking** for now
- `docs/guide/tipc.rst` + `examples/multihost/tipc_cluster/`
Two fixes fell out that are **not** TIPC-specific:
- `devx/pformat.py``pformat_caller_frame()` passed an
`indent=''` kwarg `pformat_boxed_tb()` never accepted, so every
send-side `MsgTypeError` died with a `TypeError` while
formatting itself. On `main` and every branch since
`888af602`. **Wants cherry-picking out of this stack.**
- `SpawnSpec.reg_addrs`/`.bind_addrs` pinned the wire shape to a
2-tuple. Widened to `UnwrappedAddress`, which had to become
**variadic** (`tuple[str|int, ...]`) because `msgspec` refuses
a union holding more than one array-like type.
**Acceptance bar met**: 122 passed / 1 xfailed / 2 xpassed under
`--tpt-proto tipc` across `ipc`, `discovery`, `runtime`,
`spawning`, `local`, `rpc`, `cancellation`. `tcp`/`uds`
unchanged.
[#492]: https://github.com/goodboy/tractor/pull/492
## 5. Immediate next steps (pre-land)
These live on [#493]'s body as `### TODOs before landing`. They
are **not** mirrored into an issue — if the PR is ever superseded
they need re-homing.
1. **Cherry-pick the `pformat` fix onto `main`** as its own PR —
`22ef362d` (red guard test) then `f9f98eeb` (the 1-line fix).
Unrelated to TIPC; every branch has the bug.
2. **Watch the `tipc` CI leg.** It's gated
`continue-on-error: ${{ matrix.tpt_proto == 'tipc' }}` because
GH's runners have never been asked to `modprobe` for us. Once
it has a few green runs, drop the gate. If the runners refuse
the `modprobe`, fall back to a container job with
`--cap-add NET_ADMIN`.
3. **Rebase onto `main` once #492 merges.**
## 6. The follow-up track
All filed with the `follow-up` label.
| issue | what |
| --- | --- |
| [#495] | `TIPC_IMPORTANCE` supervision QoS on the parent↔child chan |
| [#496] | `TIPC_TOP_SRV` push registry in `discovery._registry` |
| [#497] | dual-link resiliency / multi-homing |
| [#498] | `/tipc` multiaddr spec submission |
| [#499] | registrar-less discovery via name derivation |
| [#500] | multicast/group msging as a *broadcast* transport |
| [#501] | post-bind verification for instance collisions |
| [#502] | **TIPC over a `wg` mesh — the reference multihost deployment** |
### the wg direction
[#502] is the strategic one. The intent is that TIPC-over-`wg`
becomes our go-to multihost transport deployment, with composed
addresses of the form:
```
/ip4/<pub>/udp/51820/wg/u<key>/tipc/<stype>/<inst>/<scope>
\____ wg bearer ________/\_key_/\______ tractor ep ________/
```
Note the structural point, which matters for the spec proposal
in [#498]: the tcp equivalent repeats an `/ip4/…/tcp/…` inner
segment because a tcp endpoint is *located*. The tipc inner
segment has **no locative component at all** — a service name is
location-independent by design. So in the composed form the wg
segments carry all the routing and the tipc segment carries pure
*identity*.
Prerequisites already established:
- py-multiaddr [#108] (merged) proved the composed `wg` + tcp
form parses and round-trips (`['ip4','udp','wg','ip4','tcp']`)
- `examples/multihost/wg_lan/` is the existing wg example set to
generalize from rather than duplicate
- the udp-bearer-only constraint and MTU caveat are documented in
both `docs/guide/tipc.rst` and the `tipc_cluster` README
[#495]: https://github.com/goodboy/tractor/issues/495
[#496]: https://github.com/goodboy/tractor/issues/496
[#497]: https://github.com/goodboy/tractor/issues/497
[#498]: https://github.com/goodboy/tractor/issues/498
[#499]: https://github.com/goodboy/tractor/issues/499
[#500]: https://github.com/goodboy/tractor/issues/500
[#502]: https://github.com/goodboy/tractor/issues/502
[#108]: https://github.com/multiformats/py-multiaddr/pull/108
## 7. Working conventions in this repo
Provider-neutral, but they *are* enforced by review:
- **Never commit, push, rebase or amend on your own.** Prepare
changes, report them, and let the maintainer stage. Asking
"should we commit?" is a question *for you to answer*, not
permission to act.
- **A failing/guard test lands in its own commit before the fix
it guards.** Red first, then green.
- **One commit per logical step**, so history shows *why*. Never
squash unrelated changes.
- Commit subjects: present-tense verb, ~50 chars (hard max 67),
backticks around every code element. Bodies wrap at 67 cols.
- **Never write a line containing only whitespace.**
- Annotate everything including locals; prefer `match`/`case`
over `isinstance` chains; multi-line calls with trailing
commas; `@acm` over classes with `.start()`/`.stop()`.
- New modules get the `# tractor: distributed structured
concurrency.` header tagline plus the AGPL block.
- **Do not change task/checkbox state** in issues, plans or
trackers unless explicitly asked for that exact transition.
- Fix warnings at source; only genuinely-unfixable ones get
filtered, with a documented reason.
## 8. Where things are
```
tractor/ipc/_tipc.py the whole backend
tests/ipc/test_tipc.py 28 backend unit tests
tests/ipc/test_server.py the reconciliation guard
tests/devx/test_pformat.py the cherry-pick candidate
docs/guide/tipc.rst the docs page
examples/multihost/tipc_cluster/ runnable demos + manual
smoke test
ai/tpt-backends/00_shared_backend_contract.md
ai/tpt-backends/01_tipc_backend.md the (reconciled) plan
.github/workflows/ci.yml the gated tipc leg
```
Both single-host examples have been **run against a live
kernel** — the output pasted in their README is real, not
illustrative.

View File

@ -13,7 +13,7 @@ document only their own deltas.
| plan | issue | dep | size | lands | | plan | issue | dep | size | lands |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- |
| [01 — TIPC](./01_tipc_backend.md) | [#378] | **none** (stdlib) | small | first | | [01 — TIPC](./01_tipc_backend.md) | [#378] | **none** (stdlib) | small | **landed**, PR [#493] — see the [handoff](./01_tipc_HANDOFF.md) |
| [02 — QUIC/`iroh`](./02_quic_iroh_backend.md) | [#353] | `iroh` (uniffi FFI) | large | needs a prep PR | | [02 — QUIC/`iroh`](./02_quic_iroh_backend.md) | [#353] | `iroh` (uniffi FFI) | large | needs a prep PR |
| [03 — `wg` bindspace](./03_wg_tunnel_bindspace.md) | [#482], [#443] | `pyroute2` | medium, 3 layers | layer A now | | [03 — `wg` bindspace](./03_wg_tunnel_bindspace.md) | [#482], [#443] | `pyroute2` | medium, 3 layers | layer A now |
@ -52,3 +52,5 @@ lands first ships it.
[#353]: https://github.com/goodboy/tractor/issues/353 [#353]: https://github.com/goodboy/tractor/issues/353
[#482]: https://github.com/goodboy/tractor/issues/482 [#482]: https://github.com/goodboy/tractor/issues/482
[#443]: https://github.com/goodboy/tractor/issues/443 [#443]: https://github.com/goodboy/tractor/issues/443
[#493]: https://github.com/goodboy/tractor/pull/493

View File

@ -157,11 +157,40 @@ node and the client's dial keeps working, unchanged. See
(that directory is excluded from CI precisely because it needs (that directory is excluded from CI precisely because it needs
real hardware). real hardware).
Over a WireGuard mesh
~~~~~~~~~~~~~~~~~~~~~
TIPC over a `wg` mesh is the intended reference deployment for
multihost ``tractor`` (see gh #502), composing with the tunnel
examples in ``examples/multihost/wg_lan/``.
.. warning::
A wg interface is L3/``tun````POINTOPOINT,NOARP`` with
``link/none`` and no L2 address — so TIPC's ``eth`` media
**cannot** bind it. Over wg the udp bearer is *mandatory*,
not merely an alternative:
.. code:: bash
# NOT possible over wg
sudo tipc bearer enable media eth device wg0
# required instead, bound to the wg overlay IP
sudo tipc bearer enable media udp name wgmesh \
localip 10.0.11.1
Mind the MTU too: wg links typically sit at 1420, under
ethernet's 1500.
.. note:: .. note::
TIPC over a UDP bearer composes with the WireGuard tunnel TIPC is **not** unencrypted — it ships AES-GCM crypto of its
examples in ``examples/multihost/wg_lan/`` — cluster-wide own (``tipc node set key``, linux 5.9+) with cluster, master
kernel service discovery across an encrypted overlay. and per-node keys plus rekeying intervals. Those keys are
symmetric and pre-shared though, so a wg mesh is still
preferred for public-key identity, NAT traversal, and an
overlay every transport can share.
Gotchas Gotchas
------- -------
@ -198,6 +227,19 @@ protocol in the multiaddr table yet, so the grammar is
/tipc/<stype>/<instance>/<scope> /tipc/<stype>/<instance>/<scope>
Composed with a wg bearer — the form that actually matters for
multihost — that becomes:
.. code:: text
/ip4/<pub>/udp/51820/wg/u<key>/tipc/<stype>/<inst>/<scope>
Note the tipc segment carries **no** locative component, unlike
the ``/ip4/../tcp/..`` inner segment of the equivalent tcp maddr
— a TIPC service name is location-independent by design, so the
wg segments carry all the routing and the tipc segment is pure
identity.
Running the suite over TIPC Running the suite over TIPC
--------------------------- ---------------------------

View File

@ -92,8 +92,8 @@ sudo modprobe tipc
# over ethernet (L2) — simplest when the hosts share a segment # over ethernet (L2) — simplest when the hosts share a segment
sudo tipc bearer enable media eth device eth0 sudo tipc bearer enable media eth device eth0
# ..or over UDP when L2 isn't available (pairs nicely with the # ..or over UDP when L2 isn't available — and MANDATORY over a
# `wg` tunnel examples in ../wg_lan/) # `wg` mesh, see below
sudo tipc bearer enable media udp name uc localip 10.0.11.1 sudo tipc bearer enable media udp name uc localip 10.0.11.1
# verify BEFORE running anything: this must list the peer # verify BEFORE running anything: this must list the peer
@ -116,6 +116,33 @@ Both sides name the *same service*, and the kernel routes it.
Move `host_a_srv.py` to a third node and host B's dial keeps Move `host_a_srv.py` to a third node and host B's dial keeps
working, unchanged. working, unchanged.
### over a `wg` mesh
TIPC over WireGuard is the intended reference multihost
deployment (gh #502). One hard constraint: a wg interface is
L3/`tun` — `POINTOPOINT,NOARP`, `link/none`, no L2 address — so
TIPC's `eth` media **cannot** bind it. The udp bearer is
mandatory there, bound to the wg overlay IP, and wg's typical
1420 MTU sits under ethernet's 1500 so link MTU wants checking.
Composed, the deployment address is:
```
/ip4/<pub>/udp/51820/wg/u<key>/tipc/<stype>/<inst>/<scope>
\____ wg bearer ________/\_key_/\______ tractor ep ________/
```
Note the tipc segment has no locative part, unlike tcp's inner
`/ip4/../tcp/..` — a service name is location-independent, so wg
carries the routing and tipc carries identity.
Worth knowing: TIPC is **not** unencrypted. It ships AES-GCM
crypto of its own (`tipc node set key`, linux 5.9+) with
cluster/master/per-node keys and rekeying. Those keys are
symmetric and pre-shared, which is why a wg mesh is still
preferred — public-key identity, NAT traversal, and one overlay
shared by every transport.
### scope ### scope
`TIPCAddress._scope` is the backend's `.bindspace` — literally `TIPCAddress._scope` is the backend's `.bindspace` — literally